{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Convolution Nets for MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Deep Learning models can take quite a bit of time to run, particularly if GPU isn't used. \n",
    "\n",
    "In the interest of time, you could sample a subset of observations (e.g. $1000$) that are a particular number of your choice (e.g. $6$) and $1000$ observations that aren't that particular number (i.e. $\\neq 6$). \n",
    "\n",
    "We will build a model using that and see how it performs on the test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "#Import the required libraries\n",
    "import numpy as np\n",
    "np.random.seed(1338)\n",
    "\n",
    "from keras.datasets import mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Dropout, Activation, Flatten"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.layers.convolutional import Conv2D\n",
    "from keras.layers.pooling import MaxPooling2D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.utils import np_utils\n",
    "from keras.optimizers import SGD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Loading Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#Load the training and testing data\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "X_test_orig = X_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Very Important: \n",
    "When dealing with images & convolutions, it is paramount to handle `image_data_format` properly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras import backend as K"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    shape_ord = (1, img_rows, img_cols)\n",
    "else:  # channel_last\n",
    "    shape_ord = (img_rows, img_cols, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocess and Normalise Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "X_train = X_train.reshape((X_train.shape[0],) + shape_ord)\n",
    "X_test = X_test.reshape((X_test.shape[0],) + shape_ord)\n",
    "\n",
    "X_train = X_train.astype('float32')\n",
    "X_test = X_test.astype('float32')\n",
    "\n",
    "X_train /= 255\n",
    "X_test /= 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "np.random.seed(1338)  # for reproducibilty!!\n",
    "\n",
    "# Test data\n",
    "X_test = X_test.copy()\n",
    "Y = y_test.copy()\n",
    "\n",
    "# Converting the output to binary classification(Six=1,Not Six=0)\n",
    "Y_test = Y == 6\n",
    "Y_test = Y_test.astype(int)\n",
    "\n",
    "# Selecting the 5918 examples where the output is 6\n",
    "X_six = X_train[y_train == 6].copy()\n",
    "Y_six = y_train[y_train == 6].copy()\n",
    "\n",
    "# Selecting the examples where the output is not 6\n",
    "X_not_six = X_train[y_train != 6].copy()\n",
    "Y_not_six = y_train[y_train != 6].copy()\n",
    "\n",
    "# Selecting 6000 random examples from the data that \n",
    "# only contains the data where the output is not 6\n",
    "random_rows = np.random.randint(0,X_six.shape[0],6000)\n",
    "X_not_six = X_not_six[random_rows]\n",
    "Y_not_six = Y_not_six[random_rows]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Appending the data with output as 6 and data with output as <> 6\n",
    "X_train = np.append(X_six,X_not_six)\n",
    "\n",
    "# Reshaping the appended data to appropraite form\n",
    "X_train = X_train.reshape((X_six.shape[0] + X_not_six.shape[0],) + shape_ord)\n",
    "\n",
    "# Appending the labels and converting the labels to \n",
    "# binary classification(Six=1,Not Six=0)\n",
    "Y_labels = np.append(Y_six,Y_not_six)\n",
    "Y_train = Y_labels == 6 \n",
    "Y_train = Y_train.astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(11918, 28, 28, 1) (11918,) (10000, 28, 28, 1) (10000,)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape, Y_labels.shape, X_test.shape, Y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Converting the classes to its binary categorical form\n",
    "nb_classes = 2\n",
    "Y_train = np_utils.to_categorical(Y_train, nb_classes)\n",
    "Y_test = np_utils.to_categorical(Y_test, nb_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A simple CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# -- Initializing the values for the convolution neural network\n",
    "\n",
    "nb_epoch = 2  # kept very low! Please increase if you have GPU\n",
    "\n",
    "batch_size = 64\n",
    "# number of convolutional filters to use\n",
    "nb_filters = 32\n",
    "# size of pooling area for max pooling\n",
    "nb_pool = 2\n",
    "# convolution kernel size\n",
    "nb_conv = 3\n",
    "\n",
    "# Vanilla SGD\n",
    "sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Step 1: Model Definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Conv2D(nb_filters, (nb_conv, nb_conv), padding='valid', \n",
    "                 input_shape=shape_ord))  # note: the very first layer **must** always specify the input_shape\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Step 2: Compile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=sgd,\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Step 3: Fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 8s - loss: 0.2321 - acc: 0.9491 - val_loss: 0.1276 - val_acc: 0.9616\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.1065 - acc: 0.9666 - val_loss: 0.0933 - val_acc: 0.9685\n"
     ]
    }
   ],
   "source": [
    "hist = model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "                 epochs=nb_epoch, verbose=1, \n",
    "                 validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fdb2c0235f8>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VGX2wPHvSSOkQkioARIpCaGH0KSEANJEEGUVbCsW\nhLUi7IrurrK6rv52AQFFERXUtSALFkCaSpcakB5CQg819BIgBN7fHzPCJAQYwkwmM3M+zzMPmTvv\nnZwLOmfufd9zrhhjUEoppW7Ex9UBKKWUcg+aMJRSStlFE4ZSSim7aMJQSillF00YSiml7KIJQyml\nlF00YSillLKLJgyllFJ20YShlFLKLn6uDsCRIiMjTUxMjKvDUEopt7F69erDxpgoe8Z6VMKIiYkh\nNTXV1WEopZTbEJFd9o7VS1JKKaXsoglDKaWUXTRhKKWUsotHzWEopTzDhQsXyMrK4ty5c64OxWME\nBgYSHR2Nv79/kd9DE4ZSqsTJysoiNDSUmJgYRMTV4bg9YwxHjhwhKyuL2NjYIr+PXpJSSpU4586d\no1y5cposHEREKFeu3C2fsWnCUEqVSJosHMsRf5+aMIB3f8lgze5jrg5DKaVKNK9PGCdyLvDFil3c\n8/5SBk9ex6FTOsmmlLc7cuQIjRo1olGjRlSsWJEqVapcfp6bm2vXe/Tr14/09PTrjhk7dixffvml\nI0IuFmKMcXUMDpOUlGSKUul9+nwe783L5JMl2ynl58vzHWrxx9tjCPDz+nyqlEukpaVRp04dV4cB\nwLBhwwgJCWHIkCH5thtjMMbg4+M+nxOF/b2KyGpjTJI9+7vPkTpRSCk/hnaNZ+6gZJrGlOXNmWl0\nHb2IRVuzXR2aUqoEyczMJCEhgQcffJC6deuyf/9++vfvT1JSEnXr1uX111+/PLZ169asXbuWvLw8\nypQpw9ChQ2nYsCEtW7bk0KFDAPztb39j1KhRl8cPHTqUZs2aERcXx9KlSwE4c+YM9957LwkJCfTu\n3ZukpCTWrl1b/AePLqvNJzYymIn9mvFL2kFen7GZRyaspFNCBf7ePYGqEUGuDk8pr/SP6ZvYvO+k\nQ98zoXIYr91Vt0j7btmyhc8//5ykJMuX8rfffpuIiAjy8vJISUmhd+/eJCQk5NvnxIkTJCcn8/bb\nb/Piiy8yYcIEhg4detV7G2NYuXIl06ZN4/XXX2f27Nm8++67VKxYkalTp7Ju3ToSExOLFLcj6BlG\nITrUqcDcQW35c+c4FmccpsPIhYycm87Z3IuuDk0p5WI1atS4nCwAvv76axITE0lMTCQtLY3Nmzdf\ntU/p0qXp2rUrAE2aNGHnzp2Fvvc999xz1ZglS5bQp08fABo2bEjdukVLdI6gZxjXUMrPl6dTanJP\nYhXemrmFMfMymbI6i7/emUC3+hV1yZ9SxaSoZwLOEhwcfPnnjIwMRo8ezcqVKylTpgwPPfRQobUO\nAQEBl3/29fUlLy+v0PcuVarUDce4kp5h3ECl8NKM6duYyU+1JDwogKe/WsMDH60g/cApV4emlHKx\nkydPEhoaSlhYGPv372fOnDkO/x2tWrVi8uTJAGzYsKHQM5jiomcYdmoWG8GMZ1vz1crdjJibTrcx\ni3m4RXUGdaxNeFDRe7MopdxXYmIiCQkJxMfHU716dVq1auXw3/Hss8/yyCOPkJCQcPkRHh7u8N9j\nD11WWwTHzuQy4qd0vlqxmzJBAfylcxx/SKqKr49eplLKEUrSslpXy8vLIy8vj8DAQDIyMujUqRMZ\nGRn4+d389/1bXVarZxhFUDY4gH/eXZ++zaoxbNomhn67gS9X7GZYj7o0qV7W1eEppTzI6dOn6dCh\nA3l5eRhj+PDDD4uULBxBE8YtqFs5nMlPtWTaun38a2Ya936wlHsSqzC0azzlQwNdHZ5SygOUKVOG\n1atXuzoMQCe9b5mI0LNRFeYNbsfAdjWYsW4/7YcvZPyibeTmXXJ1eEop5TCaMBwkuJQfL3WJZ86g\ntjSLjeBfM7fQZfQiFmq1uFLKQ2jCcLDYyGAmPNqUCY8mcemS4Y8TVvLk56nsPpLj6tCUUuqWODVh\niEgXEUkXkUwRuaoOXkQeFJH1IrJBRJaKSEPr9qoiMl9ENovIJhF53plxOkP7+ArMGdSWl7rE82vm\nYTq+s5ARc9PJyS15xThKKWUPpyUMEfEFxgJdgQSgr4gkFBi2A0g2xtQH3gDGW7fnAYONMQlAC+Dp\nQvYt8Ur5+TKwXQ3mDW5Ht3oVeXdeJh1HLGTG+n140nJmpTxNSkrKVUV4o0aNYuDAgdfcJyQkBIB9\n+/bRu3fvQse0a9eOGy39HzVqFDk5V65IdOvWjePHj9sbulM58wyjGZBpjNlujMkFJgE9bQcYY5Ya\nY36/c9FyINq6fb8xZo3151NAGlDFibE6VcXwQEb1acz/BrSkTFAAz3z1G30/Ws6WA45tqKaUcoy+\nffsyadKkfNsmTZpE3759b7hv5cqVmTJlSpF/d8GEMXPmTMqUKVPk93MkZyaMKsAem+dZXP9D/3Fg\nVsGNIhIDNAZWFLaTiPQXkVQRSc3OLtkTzE1jIpj+bGv+eXc9thw4RbfRi3nth42cyLng6tCUUjZ6\n9+7Njz/+ePlmSTt37mTfvn00btyYDh06kJiYSP369fnhhx+u2nfnzp3Uq1cPgLNnz9KnTx/q1KlD\nr169OHv27OVxAwcOvNwW/bXXXgNgzJgx7Nu3j5SUFFJSUgCIiYnh8OHDAIwcOZJ69epRr169y23R\nd+7cSZ06dXjyySepW7cunTp1yvd7HKlE1GGISAqWhNG6wPYQYCrwgjGm0K/jxpjxWC9lJSUllfjr\nPL4+wkMtqtO9QSVGzN3Kf5fvYtq6ffy5czz3N9VqcaWuMmsoHNjg2PesWB+6vn3NlyMiImjWrBmz\nZs2iZ8+eTJo0ifvuu4/SpUvz3XffERYWxuHDh2nRogU9evS4ZjPSDz74gKCgINLS0li/fn2+1uRv\nvvkmERERXLx4kQ4dOrB+/Xqee+45Ro4cyfz584mMjMz3XqtXr2bixImsWLECYwzNmzcnOTmZsmXL\nkpGRwddff81HH33Efffdx9SpU3nooYcc83dlw5lnGHuBqjbPo63b8hGRBsDHQE9jzBGb7f5YksWX\nxphvnRinS5QJCuCNu+sx49k21CofyivfbaDn2CWs3nXU1aEppch/Wer3y1HGGF555RUaNGhAx44d\n2bt3LwcPHrzmeyxatOjyB3eDBg1o0KDB5dcmT55MYmIijRs3ZtOmTTdsKrhkyRJ69epFcHAwISEh\n3HPPPSxevBiA2NhYGjVqBFy/ffqtcuYZxiqglojEYkkUfYAHbAeISDXgW+BhY8xWm+0CfAKkGWNG\nOjFGl0uoHMY3T7Vg+vr9/OvHNO79YBn3NLZWi4dptbhS1zsTcKaePXsyaNAg1qxZQ05ODk2aNOHT\nTz8lOzub1atX4+/vT0xMTKHtzG9kx44dDB8+nFWrVlG2bFkeffTRIr3P735viw6W1ujOuiTltDMM\nY0we8AwwB8uk9WRjzCYRGSAiA6zDXgXKAe+LyFoR+X35QCvgYaC9dftaEenmrFhdTUTo0bAyvwxO\n5umUGsxYv5+U4Qv4cKFWiyvlKiEhIaSkpPDYY49dnuw+ceIE5cuXx9/fn/nz57Nr167rvkfbtm35\n6quvANi4cSPr168HLG3Rg4ODCQ8P5+DBg8yadWX6NjQ0lFOnrr59Qps2bfj+++/JycnhzJkzfPfd\nd7Rp08ZRh2sXp85hGGNmAjMLbBtn8/MTwBOF7LcE8LqL+cGl/Phz53j+0KQq//xxM2/N2sI3q/bw\n6l0JtIsr7+rwlPI6ffv2pVevXpcvTT344IPcdddd1K9fn6SkJOLj46+7/8CBA+nXrx916tShTp06\nNGnSBLDcOa9x48bEx8dTtWrVfG3R+/fvT5cuXahcuTLz58+/vD0xMZFHH32UZs2aAfDEE0/QuHFj\np11+Koy2Ny/B5qcf4vXpm9lx+Awd65Tn790TqF4u+MY7KuXmtL25c9xqe3NtDVKCpcSVZ84LbRna\nNZ5l245wx8hF/GfOFq0WV0q5hCaMEi7Az4cByTWYN6QddzaoxNj52+gwYiHT1mm1uFKqeGnCcBMV\nwgJ55/5GTBnQkojgAJ77+jfuH7+ctP1aLa48k34hcixH/H1qwnAzSTERTHumNf/qVZ+Mg6e4c8xi\nXv1hI8dzcl0dmlIOExgYyJEjRzRpOIgxhiNHjhAYeGtL9XXS240dz8ll5E9b+WL5LsJL+zOkcxx9\nmlbTanHl9i5cuEBWVtYt1Sao/AIDA4mOjsbf3z/f9puZ9NaE4QHS9p9k2LRNrNhxlLqVw/hHj7ok\nxUS4OiyllBvQVVJepk6lMCb1b8G7fRtz9Ewuvcct44VJv3HwpH47U0o5jiYMDyEi3GWtFn8mpSYz\nNxwgZfgCPliwjfN5F10dnlLKA2jC8DBBAX4M6RzHTy+25fYakfzf7C10GbWY+emHXB2aUsrNacLw\nUNXLBfPxH5P4tF9TBOg3cRWPf7qKnYfPuDo0pZSb0oTh4drFlWf2C215uWs8y7cfodM7i/j37C2c\nOa/V4kqpm6MJwwsE+PnwVHIN5g9pR/cGlXh/gaVa/Ie1e3Wdu1LKbpowvEj5sEBG3t+IqQNbEhka\nwPOT1nL/h8vZvE+rxZVSN6YJwws1qR7BD0+35q176pOZfZru7y7mb99v4NgZrRZXSl2bJgwv5esj\n9G1WjfmD2/FIyxi+WrGblBEL+GL5Li5e0stUSqmracLwcuFB/gzrUZeZz7chvmIof/t+I3e9u4RV\nO/Xe4kqp/DRhKADiK4bx9ZMteO+BxhzLyeUP45bx/KTfOHBCq8WVUhaaMNRlIkL3BpZq8Wfb12TW\nxgO0H7GA9xdkarW4Usq5CUNEuohIuohkisjQQl5/UETWi8gGEVkqIg3t3Vc5T1CAH4M7xfHzoGRa\n1Yzk37PT6fzOIuZtOejq0JRSLuS0hCEivsBYoCuQAPQVkYQCw3YAycaY+sAbwPib2Fc5WbVyQXz0\nSBKfPdYMHx/hsU9TeezTVezQanGlvJIzzzCaAZnGmO3GmFxgEtDTdoAxZqkx5pj16XIg2t59VfFJ\nrh3F7Ofb8tdudVi54yid3lnI27O0Wlwpb+PMhFEF2GPzPMu67VoeB2YVcV/lZAF+PjzZ9jbmDU6m\nR8MqjFu4jfYjFmi1uFJepERMeotICpaE8VIR9u0vIqkikpqdne344FQ+5cMCGXFfQ6YOvJ3yoYE8\nP2kt9324jE37Trg6NKWUkzkzYewFqto8j7Zuy0dEGgAfAz2NMUduZl8AY8x4Y0ySMSYpKirKIYGr\nG2tSvSw/PN2Kt++pz7bsM9z17hL++p1WiyvlyZyZMFYBtUQkVkQCgD7ANNsBIlIN+BZ42Biz9Wb2\nVa7n4yP0sakWn7RqD+2GL+C/y3ZqtbhSHshpCcMYkwc8A8wB0oDJxphNIjJARAZYh70KlAPeF5G1\nIpJ6vX2dFau6NZerxZ9rQ0KlMP7+wya6v7uEFduP3HhnpZTbEE+asExKSjKpqamuDsOrGWOYtfEA\nb/6Yxt7jZ+nRsDIvd4unUnhpV4emlCqEiKw2xiTZM7ZETHorzyEidKtfiZ9fTOa5DrWYvekA7Ycv\nZOx8rRZXyt1pwlBOUTrAlxfvqM0vLybTplYk/5mTTqd3FvFLmlaLK+WuNGEop6oaEcT4R5L4/LFm\n+PkIj3+WSr+JK9mefdrVoSmlbpImDFUs2taOYvYLbfnbnXVYtfMYnUct4q1ZaZzWanGl3IYmDFVs\n/H19eKLNbcwbkkzPRlX4cOF22g9fwHe/ZWm1uFJuQBOGKnblQwMZ/oeGfPen26kUHsigb9bRe9wy\nNu7VanGlSjJNGMplGlcry3d/asW/723AzsNnuOu9Jbzy3QaOarW4UiWSJgzlUj4+wn1NqzJvSDv6\n3R7LN6v2kDJ8AZ8v20nexUuuDk8pZUMThioRwkv78+pdCcx6vg11K4fxqrVafLlWiytVYmjCUCVK\n7QqhfPlEcz54MJFT5/LoM345z379G/uOn3V1aEp5PU0YqsQREbpaq8Wf71CLuZsO0GHEQt6bl8G5\nC1otrpSraMJQJVbpAF8G3VGbn19MJrl2FMPnbqXTO4v4afNBXYarlAtowlAlXtWIIMY93IQvHm9u\nufPf56k8OnEV27RaXKlipQlDuY3WtSKZ9Xwb/t49gTW7jtFl1CLempnGqXMXXB2aUl5BE4ZyK/6+\nPjzeOpZ5Q9pxd6MqfLhoO+1HLOTbNVlc0ps2KeVUmjCUW4oKLcV//tCQ759uReXwQF6cvI7e45Zq\ntbhSTqQJQ7m1RlXLWKrFezdg99Ec7npvCS9/u4Ejp8+7OjSlPI4mDOX2fHyE+5Is1eKPtYplcqql\nWvzTX3dotbhSDqQJQ3mMsEB//t49gdnPt6F+dDjDpm+m+7tLWLZNq8WVcgRNGMrj1KoQyhePN2fc\nQ5Zq8b4fLefpr9ZotbhSt8ipCUNEuohIuohkisjQQl6PF5FlInJeRIYUeG2QiGwSkY0i8rWIBDoz\nVuVZRIQu9Srxy+BkBnWszc+bD9J+xALe/UWrxZUqKqclDBHxBcYCXYEEoK+IJBQYdhR4DhheYN8q\n1u1Jxph6gC/Qx1mxKs8V6O/L8x1r8cvgZFLiyjPip63c8c5C5m46oNXiSt0kZ55hNAMyjTHbjTG5\nwCSgp+0AY8whY8wqoLDKKz+gtIj4AUHAPifGqjxcdNkgPnioCV8+0ZxAP1/6/3c1f5y4isxDWi2u\nlL2cmTCqAHtsnmdZt92QMWYvlrOO3cB+4IQxZm5hY0Wkv4ikikhqdnb2LYasPF2rmpHMfL4Nr3ZP\n4Lfdlmrxf2m1uFJ2KZGT3iJSFsvZSCxQGQgWkYcKG2uMGW+MSTLGJEVFRRVnmMpN+fv68FjrWOYP\nace9idF8tNhSLT51tVaLK3U9zkwYe4GqNs+jrdvs0RHYYYzJNsZcAL4FbndwfMrLRYaU4v96N+D7\nP7WiSpnSDP6fpVp8fdZxV4emVInkzISxCqglIrEiEoBl0nqanfvuBlqISJCICNABSHNSnMrLNaxa\nhm8H3s5/rNXiPcf+ytCp67VaXKkC/Jz1xsaYPBF5BpiDZZXTBGPMJhEZYH19nIhUBFKBMOCSiLwA\nJBhjVojIFGANkAf8Box3VqxK+fgIf0iqSud6FRnzcwafLt3Jjxv28+IdtXm4RXX8fEvk1VulipV4\n0tLCpKQkk5qa6uowlAfIPHSKf0zfzOKMw9SuEMKwHnW5vUakq8NSyuFEZLUxJsmesfq1SalC1Cwf\nyuePNePDh5uQk3uRBz5awdNfrmGvVosrL6YJQ6lrEBE6163Izy8m8+Idtflly0E6jFjAGK0WV15K\nE4ZSNxDo78tzHWrx84vJdIivwMifttJx5ELmaLW48jKaMJSyU3TZIMY+mMhXTzQnKMCXp/67mkcm\nrCTz0ClXh6ZUsdCEodRNur1mJDOfa8NrdyWwds9xuoxazD9nbNZqceXxNGEoVQR+vj70axXLgiHt\n6N0kmk9+3UHK8IX8L3WPVosrj6UJQ6lbUC6kFG/f24Afnm5F1YjS/HnKeu75YCnr9mi1uPI8mjCU\ncoAG0WWYOuB2hv+hIVnHznL3+7/y0pT1HNZqceVBNGEo5SA+PkLvJtHMH5LMk21uY+qaLFKGL2DC\nkh1c0HuLKw+gCUMpBwsN9OeVbnWY/UJbGlUtw+szNnPnmMUszTzs6tCUuiV2JQwRqSEipaw/txOR\n50SkjHNDU8q91SwfwuePNWP8w004e+EiD3y8goFfrCbrWI6rQ1OqSOw9w5gKXBSRmliaAFYFvnJa\nVEp5CBGhU92K/DQomcF31GZ++iE6jFjIqJ+3arW4cjv2JoxLxpg8oBfwrjHmz0Al54WllGcJ9Pfl\n2Q61+GVwOzomVGDUzxl0GLGQ2Rv3a7W4chv2JowLItIX+CMww7rN3zkhKeW5qpQpzdgHEvn6yRaE\nlPJjwBdrePiTlWQc1GpxVfLZmzD6AS2BN40xO0QkFviv88JSyrO1rFGOH59rzbC7ElifdZyuoxfz\nxozNnNRqcVWC3fT9MKz3265qjFnvnJCKTu+HodzRkdPnGT53K5NW7aZccAB/6RJP78RofHzE1aEp\nL+Dw+2GIyAIRCRORCCx3wftIREbeSpBKKYtyIaV46576THu6NdUigvjLlPX0+mApa7VaXJUw9l6S\nCjfGnATuAT43xjQHOjovLKW8T/3ocKYMuJ2R9zVk3/Gz3D32V/4yZR3Zp7RaXJUM9iYMPxGpBNzH\nlUlvpZSD+fgI9yRGM39IO55qexvf/baX9sMX8PHi7VotrlzO3oTxOjAH2GaMWSUitwEZN9pJRLqI\nSLqIZIrI0EJejxeRZSJyXkSGFHitjIhMEZEtIpImIi3tjFUptxdSyo+XrdXijauX5Z8/ptF19GKW\nZGi1uHKdm570tvuNRXyBrcAdQBawCuhrjNlsM6Y8UB24GzhmjBlu89pnwGJjzMciEgAEGWOue1FX\nJ72VJzLG8HPaId6YsZndR3PoUrcif72zDlUjglwdmvIAzpj0jhaR70TkkPUxVUSib7BbMyDTGLPd\nGJMLTAJ62g4wxhwyxqwC8q0lFJFwoC3wiXVc7o2ShVKeSkS4I6ECcwe1ZUin2izcmk3HkQt55yet\nFlfFy95LUhOBaUBl62O6ddv1VAH22DzPsm6zRyyQDUwUkd9E5GMRCbZzX6U8UqC/L8+0r8Uvg5O5\nI6ECo3+xVIvP2qDV4qp42JswoowxE40xedbHp0CUE+PyAxKBD4wxjYEzwFVzIAAi0l9EUkUkNTs7\n24khKVUyVC5TmvceSGRS/xaEBvox8Ms1PPTJCrZqtbhyMnsTxhEReUhEfK2Ph4AjN9hnL5Ymhb+L\ntm6zRxaQZYxZYX0+BUsCuYoxZrwxJskYkxQV5cwcplTJ0uK2csx4tjWv96zLxr0n6Tp6Ma9P38yJ\ns1otrpzD3oTxGJYltQeA/UBv4NEb7LMKqCUisdZJ6z5YLmvdkDHmALBHROKsmzoAm6+zi1Jeyc/X\nh0daxjB/SDvub1qViUt30H74Ar5ZtVvvLa4crsirpETkBWPMqBuM6QaMAnyBCcaYN0VkAIAxZpyI\nVARSgTDgEnAaSDDGnBSRRsDHQACwHehnjDl2vd+nq6SUt9u49wSvTdvE6l3HaBgdzrAedWlcrayr\nw1Il2M2skrqVhLHbGFOtSDs7iSYMpSzLcL9fu5e3Zm7h0Knz9G4SzUtd4okKLeXq0FQJ5PBltdf6\nPbewr1LKSUSEXo2jmTekHU8l38YPa7VaXDnGrSQMvUCqVAkWUsqPl7vWYc4LbWkSc6VafHGGriZU\nRXPdhCEip0TkZCGPU1jqMZRSJdxtUSFMfLQpn/wxiQsXL/HwJyt56r+p7Dmq9xZXN8fvei8aY0KL\nKxCllPOICB3qVKBVzUg+WbKD9+Zl0jF9IU8l12Bgcg1KB/i6OkTlBm7lkpRSys0E+vvydEpN5g1J\npnPdioz5JYOOIxcyU6vFlR00YSjlhSqFl2ZM38Z8Y60W/9OXa3jw4xWkH9BqcXVtmjCU8mLNrdXi\nb/Ssy6Z9J+k2ZjHDpm3SanFVKE0YSnk5P18fHm4Zw4Ih7ejTtCqfLdtJyvAFTFqp1eIqP00YSikA\nygYH8Gav+kx/pjU1ooIZ+u0G7n7/V9bsvm6DBeVFNGEopfKpVyWcyU+1ZHSfRhw8eY573l/K4Mnr\nOHTqnKtDUy6mCUMpdRURoWejKvwyuB0D29Vg2rq9tB++kI8WbSc3T6vFvZUmDKXUNYWU8uOlLvHM\nHZRM05iyvDkzjS6jF7Foq1aLeyNNGEqpG4qNDGZiv2ZMeDSJS5cMj0xYyZOfp7L7iFaLexNNGEop\nu7WPr8CcQW35S5c4fs08TMd3FjJibjpnc/Xe4t5AE4ZS6qaU8vPlT+1qMm9wO7rWq8i78zLpMGIB\nM9bv02pxD6cJQylVJBXDAxndpzGTn2pJeFAAz3z1G30/Ws6WAyddHZpyEk0YSqlb0iw2wlItfnc9\nthw4xZ1jlliqxXO0WtzTaMJQSt0yXx/h4RbVmT+4HQ80q8bny3aSMmIBX6/czUWtFvcYmjCUUg5T\nNjiAN+6ux/RnW1MzKoSXv93A3WN/ZfUurRb3BJowlFIOV7dyON881YLRfRqRfeo8936wlBe/Wcuh\nk1ot7s6cmjBEpIuIpItIpogMLeT1eBFZJiLnRWRIIa/7ishvIjLDmXEqpRzvSrV4Mn9qV4MZ6/eT\nMnwBHy7cptXibsppCUNEfIGxQFcgAegrIgkFhh0FngOGX+NtngfSnBWjUsr5gkv58Zcu8cwd1JYW\nt5XjrVlb6DJ6EQu1WtztOPMMoxmQaYzZbozJBSYBPW0HGGMOGWNWAVctpxCRaOBO4GMnxqiUKiYx\nkcF88mhTJj7aFGPgjxNW8sRnWi3uTpyZMKoAe2yeZ1m32WsU8BfguueuItJfRFJFJDU7W7+xKFXS\npcSXZ/YLbXipSzxLt1mqxYfPSScnN8/VoakbKJGT3iLSHThkjFl9o7HGmPHGmCRjTFJUVFQxRKeU\nulWl/HwZ2K4G84e04876lXhvfiYdRixk+jqtFi/JnJkw9gJVbZ5HW7fZoxXQQ0R2YrmU1V5EvnBs\neEopV6sQFsg79zdiyoCWlA0K4Nmvf6PP+OWk7ddq8ZLImQljFVBLRGJFJADoA0yzZ0djzMvGmGhj\nTIx1v3nGmIecF6pSypWSYiKY/mxr3uxVj60HT3HnmMW89sNGjufkujo0ZcPPWW9sjMkTkWeAOYAv\nMMEYs0lEBlhfHyciFYFUIAy4JCIvAAnGGP16oZSX8fURHmxenTvrV2LkT1v57/JdTFu3jyGd4+jT\ntBq+PuLqEL2eeNL1wqSkJJOamurqMJRSDrB530mGTd/Eyh1HqVcljH/0qEuT6hGuDsvjiMhqY0yS\nPWNL5KS3UkolVA7jm/4tGNO3MYdP5XLvB8sY9M1aDmq1uMtowlBKlVgiQo+GlZk3JJmnU2rw4/r9\ntB++gHFs8WAyAAASgElEQVRaLe4SmjCUUiVeUIAff+4cz08vtqVljXK8PWsLXUYtYn76IVeH5lU0\nYSil3Eb1csF8/MemTOzXFIB+E1fx+Ker2Hn4jIsj8w6aMJRSbiclrjyzX2jLy13jWb79CJ3eWcR/\n5mzhzHmtFncmTRhKKbcU4OfDU8k1mDekHd0bVGLs/G10GLGQH9bu1WpxJ9GEoZRyaxXCAhlprRYv\nFxLA85PWcv/45Wzep+VcjqYJQynlEZJiIpj2TGv+1as+GQdP0f3dxfz9e60WdyRNGEopj+HrIzzQ\nvBoLhqTwcIvqfLliF+2GL+CL5bv03uIOoAlDKeVxwoP8+UfPevz4XBviKoTyt+83cte7S1i186ir\nQ3NrmjCUUh6rTqUwJvVvwXsPNOZYTi5/GLeMFyb9xoETWi1eFE5rPuhWpjwOQREQWRui4iAyDkLK\ng2izM6XcnYjQvUFl2seX54MF2/hw0Xbmbj7Is+1r8VjrGEr5+bo6RLehzQcv5sGEzpCdDrmnrmwP\nDLckjqja1j/jLAmlTHXw0RMzpdzV7iM5vPHjZn7afJDYyGBe7Z5ASnx5V4flMjfTfFATxu+MgVP7\nLYkjOx0Op0P2VsufZ2xu/epXGiJr5k8iUXEQUQP8AhxzIEopp1uQfojXZ2xme/YZ2seX59XuCcRE\nBrs6rGKnCcPRco7C4a3WRLL1SkI5vvvKGPGFiNgCZyW1LQmlVKjjY1JK3bLcvEt8unQHo3/O4MJF\nw+NtYnkmpSbBpbznar0mjOKSewYOZ+RPItlb4eg2uGTToiAsukASsZ6dBEcWX6xKqWs6dPIcb8/e\nwrdr9lIhrBSvdKtDj4aVES+Yx9SE4WoXL8DRHdYEkp7/7ORCzpVxpSPyX9b6PaGERes8iVIusHrX\nMYZN28SGvSdoGlOWYT3qUrdyuKvDcipNGCXVpUtwMuvK3IhtMjlrsz7cPxgia12dTCJiwdffdfEr\n5QUuXjL8L3UP/56TzvGcXB5oXo3Bd8RRNtgz5yg1YbijM4evnmzP3mpJML/z8YeI265euRVZGwKC\nXBe7Uh7oRM4F3vnZcm/x0EA/BneK44Fmnndv8RKTMESkCzAa8AU+Nsa8XeD1eGAikAj81Rgz3Lq9\nKvA5UAEwwHhjzOgb/T63ThjXcv6U9Sxka/5kcnQHmItXxpWpdvXKrcjalvoSpVSRbTlwkmHTNrF8\n+1HqVLLcW7xZrOf8f1UiEoaI+AJbgTuALGAV0NcYs9lmTHmgOnA3cMwmYVQCKhlj1ohIKLAauNt2\n38J4ZMK4lrzzcHT71Su3DmdAnk0Va3BUISu34iCsshYmKmUnYwwzNxzgzR83s+/EOXo0rMwr3epQ\nMTzQ1aHdsptJGM5cO9YMyDTGbLcGNQnoCVz+0DfGHAIOicidtjsaY/YD+60/nxKRNKCK7b5ez68U\nlK9jedi6dNGy3Lfgyq2NU+HciSvjAkILX7lVpjr4es+SQqXsISLc2aCStVo8k3GLtvNz2kGeTqnJ\nE21ivaZa3JmfDFWAPTbPs4DmN/smIhIDNAZWOCQqT+djrQeJiIXana9sNwZOH7p6sn3bPFj31ZVx\nvgFQrmb+y1pRcVCuFvi7/7cppW5F6QBfXuwUR+8mVfnnj5v5z5x0Jqfu4dXuCXSoU8HV4Tldif4q\nKSIhwFTgBWNMoXdDEZH+QH+AatWqFWN0bkYEQitYHrFt87927sTVK7f2r4O0aWAu/f4GULY6RMVf\nvQw40LOXHSpVULVyQYx/JIlFW7MZNn0Tj3+WSkpcFH/vnsBtUSGuDs9pnDmH0RIYZozpbH3+MoAx\n5q1Cxg4DTv8+h2Hd5g/MAOYYY0ba8zu9ag6jOFw4B0cyC6zcSrdsu2hzU5qQilev3IqKg5AKOk+i\nPF5u3iU+W7qT0b9kcD7vIo+3vo1n2tckxE2qxUvKpLcflknvDsBeLJPeDxhjNhUydhg2CUMs5ZWf\nAUeNMS/Y+zs1YRSTi3lwfFfhy4DtauBYzXLpTCkPcujUOf49O50pq7OoEFaKl7vWoWejkl8tXiIS\nhjWQbsAoLMtqJxhj3hSRAQDGmHEiUhFIBcKAS8BpIAFoACwGNli3A7xijJl5vd+nCcPFbBs4Ht4K\n2Vuu0cAx0DInUnDSvVwNy2S+Um5szW5Ltfj6rBMkVbdUi9erUnIv25aYhFHcNGGUYNrAUXmRS5cM\n/1u9h3/PTudoTi59m1VjSKc4IkpgtbgmDOU+cnPgSEb+OZLs9EIaOFa5euVWpLWBYwk/5Vfe68TZ\nC4z6eSufL9tFSCk/BneqzQPNquHnW3J6xWnCUO7vmg0cM+DCmSvjSpctfOWWNnBUJUj6gVMMm7aJ\nZduPEF8xlH/0qEvz28q5OixAE4arw1DOdOkSnNx7ZZI9e8s1GjgGWRo4Fpx0j7hNGzgqlzDGMGvj\nAd78MY29x89yV8PKvNItnkrhpV0alyYM5Z3sauDoZ7k74lUNHGtBgPfdbU0Vv7O5F/lg4TbGLdyG\nrwjPtK/J461jCfR3zcpBTRhK2Tp/yuZGV1uu3cAxvFrh7VK0gaNygj1Hc3jzxzRmbzpA9XJB/P3O\nBDrUKV/sy3A1YShlj7xcy+T6jRo4BkVa5km0gaNygsUZ2fxj+mYyD50muXYUr91VvNXimjCUuhWX\nLsGJ3TZzJDaXuAo2cCzsRldlY7SBo7opFy5aq8V/zuBc3kUeax3Ls+1rFUu1uCYMpZzhWg0cD2+1\nFCz+zjfAOk9SoFVKuZrg79oJTlWyZZ86z79nb+F/q7OICi3Fy13j6dW4ilMvU2nCUKq4nTthuZSV\nvSV/Mjm+6+oGjoW1SyldxqXhq5LlN2u1+LqsEzSpXpZ/OLFaXBOGUiVFoQ0ct1qKFfM1cKxgU0ei\nDRyVpVp8yuos/m/2Fo7m5NKnaTX+3Nnx1eKaMJQq6S5dhGM7r165VbCBY6nwQlZu1bbc6EobOHqF\nE2cvMPrnDD5btpPgAF8Gd4rjweaOqxbXhKGUu7qqgaPNn2cOXRmnDRy9TsbBUwybvolfMy3V4q/d\nVZeWNW69WlwThlKeyN4GjmVjrl65FaUNHD2BMYY5mw7wxgxLtfidDSrx1251qFym6IspNGEo5U0K\na+B4eCsc2QaXLlwZF1r56pVb2sDRLZ3NvciHi7bxwYJt+IjwdEoN+retQYDfzV+m0oShlLI0cDy2\n8+qVW4U1cCxs5VZ4VW3gWMLtOZrDv2amsS37ND8+1wb/IsxraMJQSl1bwQaOtn/mHLkyThs4uo1T\n5y4QGli0f5ObSRhajqqUt/HxgTJVLY+aHfO/VlgDx11LYcNkm/39LEmj4BxJZG1t4OgiRU0WN0sT\nhlLqiuBIyyOmVf7t509bLmkVnHRPn1WggWPVAklEGzh6Ek0YSqkbKxUCVRItD1vXauC481fIO3tl\nXFBkgcl2659hVXTC3Y1owlBKFZ1fAJSvY3nYsm3gaLtya9N3cO74lXEBIdYGjvHawNENOPVfRES6\nAKMBX+BjY8zbBV6PByYCicBfjTHD7d1XKVWC+fhYPvTLxkDtTle2X6uB4/YFsO7rK+MuN3As5EZX\n2sDRZZyWMETEFxgL3AFkAatEZJoxZrPNsKPAc8DdRdhXKeVuRCC0guUR2zb/a5cbONpMuu9fD2nT\n8zdwLFOtwP1JtIFjcXHmGUYzINMYsx1ARCYBPYHLH/rGmEPAIRG582b3VUp5mMBwiE6yPGxdq4Hj\n9gVw8fyVcSEVClm5FQehFXWexEGcmTCqAHtsnmcBzR29r4j0B/oDVKtW7eajVEqVbP6BULGe5WEr\nXwNHm8tb677RBo5O4vazSsaY8cB4sBTuuTgcpVRx8fG1NFssVwPiul7Zfq0GjhlzYe0XV8b5BVpu\nalVw5Va5mtrA8RqcmTD2AlVtnkdbtzl7X6WUNxOx3G89rDLUSMn/2tljV6/c2ptqWb2F9fumWCfs\nf58f+f0SV2QtCAwr7qMpUZyZMFYBtUQkFsuHfR/ggWLYVymlCle6LFRrbnnYulYDx8yfC2ngWPDy\nVhwER3nFPInTEoYxJk9EngHmYFkaO8EYs0lEBlhfHyciFYFUIAy4JCIvAAnGmJOF7eusWJVSXi4g\nCCo1tDxsXW7gmH4lkWSnw29f5G/gGFim8JVbHtbAUZsPKqXUzbqZBo7lal69civiNkvRYwmgzQeV\nUsqZrtvA8UiBwsQtsGsZbPifzf5+UDY2/xyJGzRw1IShlFKOFFwOgm+H6rfn334zDRwLrtyKjLO8\nr4tpwlBKqeJw3QaO2/PPkfzeVj5fA8dyBVZuFX8DR00YSinlSn4BUD7e8rB1Mw0cK9aHfrOcnjg0\nYSilVEl0vQaOZ7Lzr9zKO1csZxmaMJRSyp2IQEh5yyO2TbH+as9ZIKyUUsqpNGEopZSyiyYMpZRS\ndtGEoZRSyi6aMJRSStlFE4ZSSim7aMJQSillF00YSiml7OJR7c1FJBvYVcTdI4HDDgzHHegxez5v\nO17QY75Z1Y0xUfYM9KiEcStEJNXenvCeQo/Z83nb8YIeszPpJSmllFJ20YShlFLKLpowrhjv6gBc\nQI/Z83nb8YIes9PoHIZSSim76BmGUkopu3hVwhCRLiKSLiKZIjK0kNdFRMZYX18vIomFvY87seOY\nH7Qe6wYRWSoiDV0RpyPd6JhtxjUVkTwR6V2c8TmDPccsIu1EZK2IbBKRhcUdo6PZ8d92uIhMF5F1\n1mPu54o4HUVEJojIIRHZeI3Xnf/5ZYzxigfgC2wDbgMCgHVAQoEx3YBZgAAtgBWujrsYjvl2oKz1\n567ecMw24+YBM4Hero67GP6dywCbgWrW5+VdHXcxHPMrwP9Zf44CjgIBro79Fo65LZAIbLzG607/\n/PKmM4xmQKYxZrsxJheYBPQsMKYn8LmxWA6UEZFKxR2oA93wmI0xS40xx6xPlwPRxRyjo9nz7wzw\nLDAVOFScwTmJPcf8APCtMWY3gDHG3Y/bnmM2QKiICBCCJWHkFW+YjmOMWYTlGK7F6Z9f3pQwqgB7\nbJ5nWbfd7Bh3crPH8ziWbyju7IbHLCJVgF7AB8UYlzPZ8+9cGygrIgtEZLWIPFJs0TmHPcf8HlAH\n2AdsAJ43xlwqnvBcwumfX3pPbwWAiKRgSRitXR1LMRgFvGSMuWT58ukV/IAmQAegNLBMRJYbY7a6\nNiyn6gysBdoDNYCfRGSxMeaka8NyX96UMPYCVW2eR1u33ewYd2LX8YhIA+BjoKsx5kgxxeYs9hxz\nEjDJmiwigW4ikmeM+b54QnQ4e445CzhijDkDnBGRRUBDwF0Thj3H3A9421gu8GeKyA4gHlhZPCEW\nO6d/fnnTJalVQC0RiRWRAKAPMK3AmGnAI9bVBi2AE8aY/cUdqAPd8JhFpBrwLfCwh3zbvOExG2Ni\njTExxpgYYArwJzdOFmDff9s/AK1FxE9EgoDmQFoxx+lI9hzzbixnVIhIBSAO2F6sURYvp39+ec0Z\nhjEmT0SeAeZgWWExwRizSUQGWF8fh2XFTDcgE8jB8g3Fbdl5zK8C5YD3rd+484wbN26z85g9ij3H\nbIxJE5HZwHrgEvCxMabQ5ZnuwM5/5zeAT0VkA5aVQy8ZY9y2i62IfA20AyJFJAt4DfCH4vv80kpv\npZRSdvGmS1JKKaVugSYMpZRSdtGEoZRSyi6aMJRSStlFE4ZSSim7aMJQ6gZE5KK1y+vvj2t2wC3C\ne8dcq/uoUiWN19RhKHULzhpjGrk6CKVcTc8wlCoiEdkpIv+23ktkpYjUtG6PEZF51nsS/GKtpkdE\nKojId9b7M6wTkdutb+UrIh9Z79kwV0RKW8c/JyKbre8zyUWHqdRlmjCUurHSBS5J3W/z2gljTH0s\nnVFHWbe9C3xmjGkAfAmMsW4fAyw0xjTEcl+DTdbttYCxxpi6wHHgXuv2oUBj6/sMcNbBKWUvrfRW\n6gZE5LQxJqSQ7TuB9saY7SLiDxwwxpQTkcNAJWPMBev2/caYSBHJBqKNMedt3iMG+MkYU8v6/CXA\n3xjzT2srj9PA98D3xpjTTj5Upa5LzzCUujXmGj/fjPM2P1/kytzincBYLGcjq0RE5xyVS2nCUOrW\n3G/z5zLrz0uxdE8FeBBYbP35F2AggIj4ikj4td5URHyAqsaY+cBLQDiWu8Yp5TL6jUWpGystImtt\nns82xvy+tLasiKzHcpbQ17rtWWCiiPwZyOZK19DngfEi8jiWM4mBwLXaT/sCX1iTigBjjDHHHXZE\nShWBzmEoVUTWOYwkd26ZrdTN0EtSSiml7KJnGEoppeyiZxhKKaXsoglDKaWUXTRhKKWUsosmDKWU\nUnbRhKGUUsoumjCUUkrZ5f8B7mYrlyR2uvYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdb309ee5f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8FeX1+PHPIexr2LcQEpAlYQ/XgBuKuOBKoVRBLQIq\nxSr6a2sr8rVfKHah1lpxqYqKta2K1n6xwQXqggWrLImGJSFICAhhDWuAELKd3x8zCTcRkkvI5OYm\n5/168cqdeWZ5HtB7MmdmniOqijHGGFPV6gW7A8YYY2onCzDGGGM8YQHGGGOMJyzAGGOM8YQFGGOM\nMZ6wAGOMMcYTFmCMMcZ4wgKMMcYYT1iAMcYY44n6we5AMLVr106joqKC3Q1jjAkpSUlJB1S1fUXb\n1ekAExUVRWJiYrC7YYwxIUVEvg1kO0uRGWOM8YQFGGOMMZ6wAGOMMcYTFmCMMcZ4wgKMMcYYT1iA\nMcYY4wkLMMYYYzxRp9+DMcaYOqUwH/auhx2roWVn6DfW09NZgDHGmNrq5BHIXAs7VsHO1bArCfJz\nnLb+4y3AGGOMCYAqHPnWCSbFAWX/JkBBwqDTAIibBN2GQeRwaNnF8y5ZgDHGmFDkn+7aucr5eXyv\n09aoJURc6FyhdBsGXYdCo+bV3kULMMYYEwrKS3eFR0L0CIgcBt2GQ4cYqBcW3P5iAcYYY2oeVTi8\n3QkkNSTdVRkWYIwxJtgK82HPejfV5QaU4/ucthqS7qoMCzDGGFPdTh6GnWtP3zvZlQQFJ5228EiI\nvrzGpbsqw9MAIyKjgflAGPCyqs4r094aWAj0BHKBqaq60W0LB14G+gPqtn0pIm8BfdxDhANHVHWw\niEQBm4DNbtsqVZ3u4fCMMaZipdJdXzoBJSuNknRX54EwdPLpgNKyc5A7XHU8CzAiEgY8B1wNZAJr\nRSRBVVP9NpsFJKvqWBHp624/ym2bDyxV1fEi0hBoCqCqt/qd44/AUb/jbVXVwV6NyRhjKlRuuqsV\ndLsQ+n/fCShdh0LDZsHtr4e8vIKJB9JVNQNARBYBYwD/ABMLzANQ1TQRiRKRjjhXMyOAyW5bHpDn\nf3AREeAW4EoPx2CMMeWrKN3V44rTN+Pb9w3ZdFdleBlgugI7/ZYzgWFltlkHjANWikg80B2IAAqB\nLOBVERkEJAEPquoJv30vA/ap6ha/ddEikoxzVfOoqq4s2ykRmQZMA4iMjDyP4Rlj6hxVOLyt9Lsn\nWZuctlqe7qqMYN/knwfMd4PCBuBrnOBSH4gDZqjqahGZD8wEfum370TgTb/lPUCkqh4UkaHAuyLS\nT1Wz/U+oqguABQA+n089GpcxpjYoyHNfZlx1OqCc2O+01bF0V2V4GWB2Ad38liPcdSXcL/8pUJLy\n2gZk4NxvyVTV1e6m7+AEGNxt6+Nc+Qz1O9Yp4JT7OUlEtgK9gcQqHZUxpvY6eRh2rvF7mfErv3RX\nd+g50i/dFQP1bEL68ngZYNYCvUQkGiewTABu89/AfVIsx73Hcjewwg062SKyU0T6qOpmnBv//vdu\nrgLSVDXT71jtgUOqWigiPYBeOMHKGGO+q7x0V7360Gkg+KacDigtOgW3vyHIswCjqgUicj+wDOcx\n5YWqmiIi0932F4AY4DURUSAFuMvvEDOA190nyDJwr3RcEyidHgPnoYC5IpIPFAHTVfWQB0MzxoSi\nCtNd8TDg+869k65DoWHT4Pa3FhDVunsbwufzaWKiZdCMqZW+k+5KgoJcp611lBNIim/Gt+9r6a5z\nICJJquqraLtg3+Q3xpjzpwqHMkrP3ZWV5rSVpLumOqmubsMs3VVNLMAYY0JPQR7sWef3MuOa0+mu\nxq0gIh4GjLd0V5BZgDHG1Hw5h9yp6t2pVnZ/VTrd1fNKS3edg+0HTlBQpFzQwdtJMy3AGGNqlorS\nXZ0Hge8uN6BYuutcnDhVwLPL03ll5TYu6tmW16bGe3o+CzDGmOD6TrprNZzIctoat3KCyIAfOPdP\nusRZuqsSVJV/Je/mdx9uYl/2KcbFdWXm6L6en9cCjDGmeuUccu6ZFD8qXCrdFQ0XXHX63ZN2fSzd\ndZ427jrK7IQUkr49zICurfjz7UMZ2r11tZzbAowxxjvF6S7/d08OuBU1vpPuGg4tOga3v7XIweOn\neOLf37Bo7Q7aNG3I778/gB8M7Ua9elJtfbAAY4ypOgV5sCf5dKrrTOmugbdYustDBYVF/H3Vtzz5\n0TecyCtk6iXRPDCqF62aNKj2vliAMcZUXtl0164kKDzltFm6q9p9kX6AXy1JZfO+Y1xyQVvm3NSP\nXh1bBK0/FmCMMYEpN93VwEl3xd/jBJRuwyzdVY0yD+fw2w828cGGvUS0bsILdwzl2n4dceYQDh4L\nMMaYMys45TzddcZ0V7gTRAbd6r7MGAcNmgS3v3VQbn4hL/xnK89/thUR+OnVvZk2ogeNG9SMomYW\nYIwxjpxDpd892fVVmXTX1advxrfrbemuIFJVlm7cy6/f38SuIye5YWBnZl0fQ9fwmhXkLcAYUxep\nwsGtpd89OfCN01Y23RU5HJp3CG5/TYlv9h1jTkIKX2w9SN9OLXjznuFc1LNtsLt1RhZgjKkLCk7B\n7uTT9052roacA05bSbprgqW7arCjOfn86eNv+Nuqb2neqD5zx/TjtvhI6ofV3CtJCzDG1EYnDrr3\nTYpfZvz6dLqrTQ/odY2lu0JEYZHyj8SdPL5sM4dz8rgtPpKfXdOHNs0aBrtrFbIAY0yoqyjd1WWw\nk+4qnqre0l0hI+nbQ8xJSGXDrqNcGNWa2TfF079rq2B3K2CeBhgRGQ3Mx6lo+bKqzivT3hpYCPQE\ncoGpqrrRbQsHXgb6A+q2fSkic4B7APdxFmap6gfuPo/gVMUsBB5Q1WVejs+YoCgv3dWktZvumui+\nzDjE0l0haF92Lr//MI3/+3oXnVo2Zv6Ewdw8qEvQHzs+V54FGBEJA54DrgYygbUikqCqqX6bzQKS\nVXWsiPR1tx/lts0HlqrqeLdssv8rv39S1SfKnC8Wp5RyP6AL8LGI9FbVQi/GZ0y1CTTdFXkRtO1l\n6a4QdqqgkFf/u51nPtlCfqHy4yt6ct/IC2jWKDSTTV72Oh5IV9UMABFZBIwB/ANMLDAPQFXTRCRK\nRDriXM2MACa7bXlAXgXnGwMsUtVTwDYRSXf78GWVjcgYr6nCwfTSLzMe3OK0WbqrVluetp+576Wy\n7cAJrorpwKM3xBLVrlmwu3VevAwwXYGdfsuZwLAy26wDxgErRSQe6A5E4KS4soBXRWQQkAQ8qKon\n3P1miMgkIBH4maoeds+3qsz5ulbtkIypYgWnnCsS/5cZcw46bcXprsG3WbqrFtt24ASPvZfKp2n7\n6dGuGa9OuZCRfWrHLw7Bvu6aB8wXkWRgA/A1TnCpD8QBM1R1tYjMB2YCvwSeBx7DuS/zGPBHYGqg\nJxSRacA0gMjIyKobiTGBKE537fjS+bn7ayh0L87b9ITeo0+/e2Lprlrt+KkCnv00nVc+z6BR/TBm\nXd+XyRdH07B+7fk39zLA7AK6+S1HuOtKqGo2MAVAnLtX24AMnPstmaq62t30HZwAg6ruK95fRF4C\n3gv0fO7+C4AFAD6fTys3NGMCUGG6awgM+5HzqHC3YdC8fXD7a6qFqvJu8i5+90Ea+4+dYvzQCH4x\nug8dWjQOdteqnJcBZi3QS0Sicb7oJwC3+W/gPimW495juRtY4QadbBHZKSJ9VHUzzo3/VHefzqq6\nxz3EWGCj+zkBeENEnsS5yd8LWOPh+IwpLT/3u1PVl0p3DYchtzs/uwyBBrXvC8WUb0PmUeYscYp/\nDYxoxQs/HEpcZPUU/woGzwKMqhaIyP3AMpzHlBeqaoqITHfbXwBigNdERIEUnEeMi80AXnefIMvA\nvdIBHheRwTgpsu3Aj9zjpYjI2ziBqAC4z54gM546caD03F2W7jJn4RT/2syitTtp07Qhj39/IOOH\nRlRr8a9gENW6myXy+XyamJgY7G6YUKAKB7b4vXuyykl/AYQ1hM6DT78Zb+ku4/Iv/pWTV8idF0cF\nrfhXVRKRJFX1VbRdsG/yG1Mz5ec6VyT+LzOePOS0NWnjBJEhd1i6y5zVF+kHmLMkhW/2HefSC9ox\n+6bYoBb/CgYLMMZA+emuthdAn+v95u7qBSH2RrWpPjsPOcW/PtzoFP968YdDuSY2+MW/gsECjKl7\nKkp3dRkCw6affpmxWbvg9teEhLLFv352dW/uqUHFv4LBAoyp/QJNd0Ve5NxLsXSXOQdli3/dOLAz\nj9TA4l/BYAHG1D4nDpR+92RPsqW7jCc27z3Gr5acLv61aNpwhveomcW/gsECjAltqs7U9MX3Tnas\ngkNbnTZLdxmPlC3+9diYfkys4cW/gsECjAktxemu4qlWdq6Gk4edtiZtnEASN8n5aekuU8UKi5S3\nE3fyB7/iXw9d04fWIVD8KxgswJia7XhW6anqS6W7ekHfG5xUV+RwJ/1l6S7jkaRvDzE7IYWNu7JD\nsvhXMFiAMTVHhemuOBh+7+mXGZtZrtt4b192LvM+TGNxiBf/CgYLMCZ48nNh91el5+4qTnc1besE\nEUt3mSA5VVDIws+388ynWygoVO4feQH3XtEzZIt/BYP9TZnqczyrdN343clQlO+0WbrL1CCfpu1j\n7pJUth/M4aqYjvzyxhi6tw3t4l/BYAHGeKOoyEl3+b/MeCjDaStOd130Y0t3mRolI+s4j72XyvLN\nWfRo34y/TLmQK2pJ8a9gsABjqkb+ye9WZiyV7hoOQye7c3cNhvqNgtpdY/yVLf71P9fHcOfFUbWq\n+FcwWIAxlVNeuqtdb+h7o/vuyXBo29PSXaZGqkvFv4LBAoypWLnprkbOy4wX/diZaiUi3tJdJiRs\nyDzK7ISNfLXjCIMiWvHiD4cypBYX/woGCzDmu/JPwq6vSs/dlXvEabN0lwlx/sW/2jZryOPjBzI+\nrvYX/woGCzAGju8v/e7JnnWl010xN1m6y4S8fL/iXyfzCrnrkmgeuKoXLRuHdvGvmszTACMio4H5\nOCWTX1bVeWXaWwMLgZ5ALjBVVTe6beHAy0B/nPLIU1X1SxH5A3ATkAdsBaao6hERiQI2AZvdw69S\n1eleji8kFae7iqda2bEKDm9z2sIaQdc4uOi+03N3NW0T3P4aUwX+m36AX7nFvy7r5RT/uqBD3Sr+\nFQyeBRgRCQOeA64GMoG1IpKgqql+m80CklV1rIj0dbcf5bbNB5aq6ngRaQg0ddd/BDyiqgUi8nvg\nEeBht22rqg72akwhqdx0VzsnkPimui8zDrJ0l6lV/It/dWvThAU/HMrVdbT4VzB4eQUTD6SragaA\niCwCxgD+ASYWmAegqmkiEiUiHXGuZkYAk922PJwrFlT13377rwLGeziG0FNuuqsPxN58+mXGNj0s\n3WVqpZN5TvGvF/5jxb+CycsA0xXY6becCQwrs806YBywUkTige5ABFAIZAGvisggIAl4UFVPlNl/\nKvCW33K0iCQDR4FHVXVl2U6JyDRgGkBkZGQlh1ZDFBXBgc2lA4qlu0wdpqp8uHEvv/Er/jXr+hi6\nWPGvoAj2Tf55wHw3KGwAvsYJLvWBOGCGqq4WkfnATOCXxTuKyP8ABcDr7qo9QKSqHhSRocC7ItJP\nVbP9T6iqC4AFAD6fTz0dXVXLyykzd9caS3cZ49q89xhzElL4MsOKf9UUXgaYXUA3v+UId10J98t/\nCoA4SdFtQAbO/ZZMVV3tbvoOToDB3XYycCMwSlXVPdYp4JT7OUlEtgK9gcSqHli1Obav9Lsne9ZB\nUYHTZukuYwAr/lWTeRlg1gK9RCQaJ7BMAG7z38B9UizHvcdyN7DCDTrZIrJTRPqo6macG/+p7j6j\ngV8Al6tqjt+x2gOHVLVQRHoAvXCCVWj4TrrrSzi83Wmr39iZu+viGe7cXfGW7jJ1XmGR8tbanfxh\nWRpHT+Zz27BIfna1Ff+qSTwLMO5TXvcDy3AeU16oqikiMt1tfwGIAV4TEQVSgLv8DjEDeN19giwD\n90oHeBZoBHzkPglS/DjyCGCuiOQDRcB0VT3k1fjO23fSXash96jT1qy9c8/kwrudgNJ5ENS3/2mM\nKZa4/RBzljjFv+Kj2jD75lj6dbHiXzWNuBmmOsnn82liYjVl0MpLd7Xv6wSU4pvxlu4y5ozKFv+a\ndUMMNw3sbI8dVzMRSVJVX0XbBfsmf+1UVARZaaUDin+6q+tQuPgBJ6BEXGjpLmMqcKbiXz8e2ZOm\nDe0rrCazf52qkJcDu5JOB5TMNZbuMqaK+Bf/ujq2I4/eYMW/QoUFmMrIPQoZnzn3T3asgr3rS6e7\nYr9n6S5jzlPZ4l+vTY3n8t7tg90tcw4swFTGgS3w9iRLdxnjgeOnCnjm0y0s/HwbjeqH8egNMUy6\nyIp/hSILMJXRaSDc/Ynz09JdxlSJoiK3+NeHaWQdO8UPhkbwcyv+FdIswFRG/YYQUeEDFMaYAK3P\nPMKchBSn+Fe3cBZY8a9awQKMMSZoDhw/xRPLNvNWohX/qo0swBhjql1+YRF/+/Jb/vSxFf+qzSzA\nGGOq1X/TDzAnIYUt+634V21nAcYYUy12HsrhN+9vYmmKFf+qKyzAGGM8dTKvkOf/s5UX/7OVeiI8\ndE1v7r7Min/VBRUGGBGZAfxdVQ9XQ3+MMbVE2eJfNw3qwiPX9bXiX3VIIFcwHYG1IvIVsBBYpnV5\nhkxjTIXS9mbzq4TUkuJfb00bzjAr/lXnVBhgVPVREfklcA3OlPnPisjbwCuqutXrDhpjQod/8a8W\njevz2Pf6M/HCblb8q44K6B6MqqqI7AX24pQpbg28IyIfqeovvOygMabmK1v86/Zh3fnp1b2t+Fcd\nF8g9mAeBScAB4GXg56qaLyL1gC041SWNMXVU4vZDzE5IIWV3NvHRbZhzUz9iu7QMdrdMDRDIdWsb\nYJyqXquq/1DVfABVLQJuLG9HERktIptFJF1EZp6hvbWILBaR9SKyRkT6+7WFi8g7IpImIptE5CJ3\nfRsR+UhEtrg/W/vt84h7rs0icm2AfwfGmErYezSX/7foa8a/8CWHTuTxzMQhvDVtuAUXUyKQFNmH\nQEnpYRFpCcSo6mpV3XS2nUQkDHgOuBrIxHlQIEFVU/02mwUkq+pYEenrbj/KbZsPLFXV8W7Z5Kbu\n+pnAJ6o6zw1aM4GHRSQWmAD0A7oAH4tIb1UtDGCMxpgAnSoo5JXPt/Hsp+kUFCkzrryAe6+w4l/m\nuwL5L+J5IM5v+fgZ1p1JPJCuqhkAIrIIGAP4B5hYYB6AqqaJSJSIdARygRHAZLctD8hz9xkDXOF+\nfg34DHjYXb9IVU8B20Qk3e3DlwGM0RgTgE827WPue6l8ezCHa2I78ugNsUS2bVrxjqZOCiTAiP9j\nyapaJCKB7NcV2Om3nAkMK7PNOmAcsFJE4oHuQARQCGQBr4rIICAJeFBVTwAdVXWPu/9enMeoi8+3\nqsz5ugbQT2NMBTKyjjP3vVQ+c4t//XVqPCOs+JepQCD3YDJE5AERaeD+eRDIqKLzzwPCRSQZmAF8\njRNc6uNcIT2vqkOAEzipsFLcwHdO7+SIyDQRSRSRxKysrPPtvzG12rHcfH73wSaufWoFidsP8+gN\nMSx9cIQFFxOQQK5EpgNPA4/ifJl/AkwLYL9dQDe/5Qh3XQlVzcZ5twZxJiTahhO8mgKZqrra3fQd\nTgeYfSLSWVX3iEhnYH+g53PPuQBYAODz+eyFUWPOoKhIWfz1LuYtteJfpvICedFyP87N83O1Fugl\nItE4X/QTgNv8NxCRcCDHvcdyN7DCDTrZIrJTRPqo6macG//F924SgDtxrn7uBP7lt/4NEXkS5yZ/\nL2BNJfptTJ22PvMIsxNS+Not/vXSJB+Du4UHu1smBAXyHkxj4C6cp7NKfn1R1anl7aeqBSJyP7AM\nCAMWqmqKiEx3218AYoDXRESBFPc8xWYAr7tPkGXgXungBJa3ReQu4FvgFvd4Ke4MA6k4L4PeZ0+Q\nGRO4A8dP8Yelm3k7aSdtmzXiD+MH8n0r/mXOg1Q0rZiI/ANIw7n6mAvcDmxS1Qe97563fD6fJiYm\nBrsbxgRV2eJfUy6JYsYoK/5lzk5EklS1wrrxgdyDuUBVfyAiY1T1NRF5A1h5/l00xgTb51sO8Ksl\n/sW/+nFBh+bB7papJQIJMPnuzyPum/Z7gQ7edckY4zX/4l+RbZry0iQfV8V0sOJfpkoFEmAWuNOx\nPIpzI7058EtPe2WM8UTZ4l8/v7YPd10abcW/jCfKDTDuhJbZbrGxFUCPaumVMaZKqSofbNjLb95P\nZffRXG4e1IVHru9L51ZW/Mt4p9wA4761/wvg7WrqjzGmiqXtzWZOQgqrMg4R07klf7p1sBX/MtUi\nkBTZxyLyEPAWzhv1AKjqobPvYowJtiM5efzpI6f4V8smDfj19/ozMT6SMHvs2FSTQALMre7P+/zW\nKZYuM6ZGKixSFq3dwRPLNnP0ZD53DHeKf4U3teJfpnoF8iZ/dHV0xBhz/vyLfw2LbsOcm/sR09nq\ns5jgCORN/klnWq+qf6367hhjKmPv0Vx+9+Em/pW8m86tGvPMxCHcOLCzPXZsgiqQFNmFfp8b48wL\n9hVgAcaYIDtVUMjLK7fx3HIr/mVqnkBSZDP8l90JKhd51iNjTIVUlU/T9lvxL1OjVebXnBOA3Zcx\nJki2Zh3nMbf4V08r/mVqsEDuwSzhdFGvejhlju29GGOq2bHcfJ79NJ2F/91G4/phPHpDDHdeHEWD\nsEDqBhpT/QK5gnnC73MB8K2qZnrUH2NMGWWLf93ii+Dn1/alfYtGwe6aMeUKJMDsAPaoai6AiDQR\nkShV3e5pz4wxrNt5hDlLnOJfg634lwkxgQSYfwAX+y0XuusuPPPmxpjzVbb41xM/GMS4IV2t+JcJ\nKYEEmPpuSWMAVDXPrTJZIREZDczHqWj5sqrOK9PeGlgI9ARygamqutFt2w4cwwloBcXFbUTkLaCP\ne4hw4IiqDhaRKGATsNltW6Wq0wPppzE1RX5hEX/98luecot/3XNZD2ZceQEtrPiXCUGBBJgsEblZ\nVRMARGQMcKCinUQkDHgOuBrIBNaKSIKqpvptNgtIVtWxItLX3X6UX/tIVS11LlUtnroGEfkjcNSv\neauqDg5gTMbUOJ9vOcCcJSmk7z/OiN7t+d8bY634lwlpgQSY6cDrIvKsu5wJnPHt/jLigXRVzQAQ\nkUXAGMA/wMQC8wBUNU1EokSko6ruq+jg4ryifAtwZQB9MabG2nkoh1+/n8qylH1EtmnKy5N8jLLi\nX6YWCORFy63AcBFp7i4fD/DYXYGdfsuZwLAy26wDxgErRSQe6A5EAPtwHo3+WEQKgRdVdUGZfS8D\n9qnqFr910SKSjHNV86iqWmlnU2OdzCvk+c/SeWFFBmFW/MvUQoG8B/Nb4HFVPeIutwZ+pqqPVsH5\n5wHz3aCwAfga554LwKWquktEOgAfiUiaqq7w23ci8Kbf8h4gUlUPishQ4F0R6aeq2WXGMw2YBhAZ\nGVkFQzDm3JQt/jVmcBdmXmfFv0ztE0iK7DpVnVW8oKqHReR6nBLK5dkFdPNbjnDXlXC//KdAScpr\nG5Dhtu1yf+4XkcU4KbcV7rb1ca58hvod6xRwyv2cJCJbgd5AYplzLgAWAPh8PsWYalS2+NdTE4YQ\nH90m2N0yxhOBBJgwEWnkfoEjIk2AQN7wWgv0EpFonMAyAbjNfwN3XrMc9ym1u4EVqpotIs2Aeqp6\nzP18DTDXb9ergDT/Fz5FpD1wSFULRaQH0As3WBkTbEdy8njyo2/4uxX/MnVIIAHmdeATEXkVEGAy\n8FpFO6lqgYjcDyzDeUx5oaqmiMh0t/0FIAZ4TUQUSAHucnfvCCx2b3LWB95Q1aV+h59A6fQYwAhg\nrojkA0XAdKu6aYLNin+ZukxUK84Sue+zXIVz4z0b6KSq95W/V83n8/k0MTGx4g2NqYS12w8x+18p\npO6x4l+mdhGRpOJ3E8sT6GzKxU91/QDnPsk/z6NvxtRqZYt/PXvbEG4YYMW/TN1z1gAjIr1xntSa\niPNi5Vs4Vzwjq6lvxoSU3PxCXvn8dPGvB668gOlW/MvUYeX9l58GrARuVNV0ABH5SbX0ypgQoqp8\nsmk/j73vFP+6tp9T/KtbGyv+Zeq28gLMOJyb6ctFZClOFUu7xjfGz9as48xdksp/vsnigg7N+dtd\n8VzWy4p/GQPlBBhVfRfnZcVmOFO8/D+gg4g8DyxW1X9XUx+NqXGO5ebzzKfpLPx8G00ahPHLG2OZ\ndFF3K/5ljJ9Apoo5AbwBvOG+xf8D4GHAAoypc4qKlP/7ehfzPkzj4IlT3DK0Gz8f3Yd2za34lzFl\nndPdR1U9jPMWfNl5wYyp9dbtPMLshBSSdzrFv16508cgK/5lzFnZ4y3GVCDr2Cn+sCyNtxMzade8\nEX/8wSDGWvEvYypkAcaYsygp/vXRN+QWFPKjET2434p/GRMwCzDGnMHKLVn8aklqSfGv2TfF0rO9\nFf8y5lxYgDHGz46DTvGvf6da8S9jzpcFGGOw4l/GeMECjKnTVJX3N+zht+9vsuJfxlQxCzCmztq0\nxyn+tXrbIWI7t2T+xCFcGGXFv4ypKhZgTJ3jX/yrVZMG/GZsfyZcaMW/jKlqFmBMnVFYpLy5ZgdP\n/Hsz2Sfz+eHw7vzEin8Z4xkLMKZOWLPtEHMSnOJfw3u0YfZNVvzLGK95OjOfiIwWkc0iki4iM8/Q\n3lpEFovIehFZIyL9/dq2i8gGEUkWkUS/9XNEZJe7PllErvdre8Q912YRudbLsZnQsOfoSR5482tu\nefFLjuTk8dxtcbx5z3ALLsZUA8+uYEQkDHgOuBrIBNaKSIKqpvptNgtIVtWxItLX3X6UX/tIVT1w\nhsP/SVWfKHO+WJzyAv2ALsDHItJbVQurblQmVBQX/3r203QKVXlgVC/uvbwnTRraY8fGVBcvU2Tx\nQLqqZgCIyCKcaf/9A0wsMA9AVdNEJEpEOqrqvkqcbwywSFVPAdtEJN3tw5fnMwgTWlSVjzft57H3\nUtlxyIpyEvrBAAAXIElEQVR/GRNMXqbIugI7/ZYz3XX+1uEUNkNE4oHuQITbpjhXIUkiMq3MfjPc\ntNpCt4RAoOcztdjWrONMfnUt9/w1kYb16/H3u4bx4g99FlyMCZJg3+SfB8wXkWRgA/A1UJzSulRV\nd4lIB+AjEUlT1RXA88BjOAHoMeCPwNRAT+gGq2kAkZGRVTYQEzylin81DON/b4zlh1b8y5ig8zLA\n7AK6+S1HuOtKqGo2MAVAnMmetgEZbtsu9+d+EVmMk+5a4Z8+E5GXgPcCPZ97vJJ6Nj6fTys/PBNs\nVvzLmJrNywCzFuglItE4X/QTgNv8NxCRcCBHVfOAu3ECSLZbprmeqh5zP18DzHX36ayqe9xDjAU2\nup8TcKpuPolzk78XsMbD8Zkg8i/+NSTSin8ZUxN5FmBUtUBE7geWAWHAQlVNEZHpbvsLQAzwmogo\nkALc5e7eEVjszmBbH3hDVZe6bY+LyGCcFNl24Efu8VJE5G2chwgKgPvsCbLax7/4V/sWVvzLmJpM\nVOtulsjn82liYmLFG5qgyy8s4rUvtjP/4y3kFhQy9ZJoK/5lTJCISJKq+iraLtg3+Y2pkH/xr8t7\nt+d/rfiXMSHBAoypsfyLf3Vv25RX7vRxZV8r/mVMqLAAY2qcnLwCnv9sKy+uyKB+PeEXo53iX43q\n21v4xoQSCzCmxlBV3lu/h99+sIk9R3P53uAuzLwuhk6tGge7a8aYSrAAY2qE1N3ZzFmSwppth+jX\npSVPW/EvY0KeBRgTVIdPOMW/Xl/tFP/67dgB3HphNyv+ZUwtYAHGBEVhkfLGmh380Yp/GVNrWYAx\n1W7NtkPMTkhhk1v8a87N/ejbyeqzGFPbWIAx1WbP0ZP87oM0Etbtpmt4E/58exzX9e9kjx0bU0tZ\ngDGe8y/+VaTKg6N6Md2KfxlT61mAMZ4pW/xrdL9O/M8NMVafxZg6wgKM8UT6/uPMfS+VFd9k0atD\nc/5+1zAu7dUu2N0yxlQjCzCmSh3LzefpT7bw6n+3W/EvY+o4CzCmShQVKf/8KpPfL93MwROnuNXX\njYeuteJfxtRlFmDMeUt2i3+tc4t/LZzsY2CEFf8ypq6zAGMqLevYKR5fmsY/kpziX0/eMojvDbbi\nX8YYh6cBRkRGA/NxKlq+rKrzyrS3BhYCPYFcYKqqbnTbtgPHgEKgoLi4jYj8AbgJyAO2AlNU9YiI\nRAGbgM3u4Vep6nQvx1dXlS3+9aPLezDjyl40b2S/rxhjTvPsG0FEwoDngKuBTGCtiCSoaqrfZrOA\nZFUdKyJ93e1H+bWPVNUDZQ79EfCIW5L598AjwMNu21ZVHezFeIxjxTdZ/GpJCluzTnBFn/b8742x\n9LDiX8aYM/DyV854IF1VMwBEZBEwBvAPMLHAPABVTRORKBHpqKr7znZQVf233+IqYHyV99x8x46D\nOTz2fiofpe4jqm1TFk72cWXfjsHuljGmBvMywHQFdvotZwLDymyzDhgHrBSReKA7EAHsAxT4WEQK\ngRdVdcEZzjEVeMtvOVpEkoGjwKOqurJKRlKH5eQV8OflW1mw0op/GWPOTbCT5vOA+W5Q2AB8jXPP\nBeBSVd0lIh2Aj0QkTVVXFO8oIv8DFACvu6v2AJGqelBEhgLvikg/Vc32P6GITAOmAURGRno5tpBm\nxb+MMefLywCzC+jmtxzhrivhfvlPARBnxsNtQIbbtsv9uV9EFuOk3Fa4204GbgRGqaq6250CTrmf\nk0RkK9AbSCxzzgXAAgCfz6dVNtpapGzxr2cmDsFnxb+MMefIywCzFuglItE4gWUCcJv/BiISDuSo\nah5wN7BCVbNFpBlQT1WPuZ+vAea6+4wGfgFcrqo5fsdqDxxS1UIR6QH0wg1WJjD+xb/Cmzbkd+MG\ncIvPin8ZYyrHswDjPuV1P7AM5zHlhaqaIiLT3fYXgBjgNRFRIAW4y929I7DYnca9PvCGqi51254F\nGuGkzeD048gjgLkikg8UAdNV9ZBX46tN/It/HcstYNJFUfzkqt60atog2F0zxoQwcTNMdZLP59PE\nxMSKN6zFVmccZM6SVDbtyeaiHm2ZfXOsFf8yxpRLRJKK300sT7Bv8psg2X3kJL/7MI0lVvzLGOMR\nCzB1TG5+IS+vzOC55Vut+JcxxlMWYOoIVeWj1H089n4qOw+d5Lr+nZh1vRX/MrVHfn4+mZmZ5Obm\nBrsrtUbjxo2JiIigQYPK3Y+1AFMHlC3+9frdw7jkAiv+ZWqXzMxMWrRoQVRUlKV6q4CqcvDgQTIz\nM4mOjq7UMSzA1GLZufk8/fEW/vKFU/xr9k2x3DHcin+Z2ik3N9eCSxUSEdq2bUtWVlalj2EBphYq\nKlLe+SqTx5emcfBEHhMu7MZD1/ShrRX/MrWcBZeqdb5/nxZgahn/4l9xkeG8OjmeARGtgt0tY2q9\ngwcPMmqUMxn83r17CQsLo3379gCsWbOGhg0bVniMKVOmMHPmTPr06XPWbZ577jnCw8O5/fbbq6bj\nHrIAU0v4F//q0KIRf7rVKf5lv9EZUz3atm1LcnIyAHPmzKF58+Y89NBDpbZRVVSVevXOnKZ+9dVX\nKzzPfffdd/6drSaWjA9xeQVFvLwygyuf+Ix3k3fxo8t78OlDVzB2SIQFF2NqgPT0dGJjY7n99tvp\n168fe/bsYdq0afh8Pvr168fcuXNLtr300ktJTk6moKCA8PBwZs6cyaBBg7jooovYv38/AI8++ihP\nPfVUyfYzZ84kPj6ePn368MUXXwBw4sQJvv/97xMbG8v48ePx+Xwlwa862RVMCPMv/jWyT3t+acW/\njAHgV0tSSN2dXfGG5yC2S0tm39SvUvumpaXx17/+FZ/Pefl93rx5tGnThoKCAkaOHMn48eOJjY0t\ntc/Ro0e5/PLLmTdvHj/96U9ZuHAhM2fO/M6xVZU1a9aQkJDA3LlzWbp0Kc888wydOnXin//8J+vW\nrSMuLq5S/T5fdgUTgnYczOGevyYyaeEaCouUhZN9vDol3oKLMTVUz549S4ILwJtvvklcXBxxcXFs\n2rSJ1NTU7+zTpEkTrrvuOgCGDh3K9u3bz3jscePGfWebzz//nAkTJgAwaNAg+vWrXGA8X3YFE0LK\nFv96eHRfpl4aZcW/jCmjslcaXmnWrFnJ5y1btjB//nzWrFlDeHg4d9xxxxlfDvV/KCAsLIyCgoIz\nHrtRo0YVbhMsdgUTAlSVhHW7GfXH//Ds8nRuGNCZ5Q9dwb1X9LTgYkyIyc7OpkWLFrRs2ZI9e/aw\nbNmyKj/HJZdcwttvvw3Ahg0bzniFVB3sCqaGS92dzZyEFNZsP0T/rlb8y5hQFxcXR2xsLH379qV7\n9+5ccsklVX6OGTNmMGnSJGJjY0v+tGpV/a8r2HT9NXS6/sMn8vjjR5t5Y/UOwps25OfX9rHiX8aU\nY9OmTcTExAS7GzVCQUEBBQUFNG7cmC1btnDNNdewZcsW6tc/92uKM/292nT9IaqwSHlj9bc88e9v\nOH7Kin8ZY87d8ePHGTVqFAUFBagqL774YqWCy/myAFODlC3+NefmfvTp1CLY3TLGhJjw8HCSkpKC\n3Q1vb/KLyGgR2Swi6SLynQe4RaS1iCwWkfUiskZE+vu1bReRDSKSLCKJfuvbiMhHIrLF/dnar+0R\n91ybReRaL8dWlXYfOcmMN7/m1gWryD6Zz/O3x/HGPcMsuBhjQppnVzAiEgY8B1wNZAJrRSRBVf0f\nZ5gFJKvqWBHp624/yq99pKoeKHPomcAnqjrPDVozgYdFJBaYAPQDugAfi0hvVS30ZIBVIDe/kJdW\nZPDnz5ziX//vql78aIQV/zLG1A5epsjigXRVzQAQkUXAGMA/wMQC8wBUNU1EokSko6ruK+e4Y4Ar\n3M+vAZ8BD7vrF6nqKWCbiKS7ffiyykZURcoW/7p+gFP8K6K1Ff8yxtQeXgaYrsBOv+VMYFiZbdYB\n44CVIhIPdAcigH2A4lyFFAIvquoCd5+OqrrH/bwX6Oh3vlVlzte1bKdEZBowDSAyMrJyIzsP6fuP\n8aslqazccoDeHZvzxt3DuNiKfxljaqFgv2g5DwgXkWRgBvA1UJzSulRVBwPXAfeJyIiyO6vzjPU5\nPWetqgtU1aeqvuKptKtDdm4+v34vldFPrXSm1L8plvcfuMyCizG1xMiRI7/z0uRTTz3Fvffee9Z9\nmjd3pnfavXs348ePP+M2V1xxBRW9TvHUU0+Rk5NTsnz99ddz5MiRQLvuGS8DzC6gm99yhLuuhKpm\nq+oUN5BMAtoDGW7bLvfnfmAxTroLYJ+IdAZwf+4P9HzBUFSkvJ24kyuf+IxX/ruNH/gi+OyhK5hy\nSbRVljSmFpk4cSKLFi0qtW7RokVMnDixwn27dOnCO++8U+lzlw0wH3zwAeHh4ZU+XlXx8htuLdBL\nRKJFpCHODfgE/w1EJNxtA7gbWKGq2SLSTERauNs0A64BNrrbJQB3up/vBP7lt36CiDQSkWigF7DG\no7EF5Osdhxn75//yi3fWE9mmKQn3Xcrvxg20ypLG1ELjx4/n/fffJy8vD4Dt27eze/duhgwZwqhR\no4iLi2PAgAH861//+s6+27dvp39/5yHakydPMmHCBGJiYhg7diwnT54s2e7ee+8tmeZ/9uzZADz9\n9NPs3r2bkSNHMnLkSACioqI4cMB5PurJJ5+kf//+9O/fv2Sa/+3btxMTE8M999xDv379uOaaa0qd\np6p4dg9GVQtE5H5gGRAGLFTVFBGZ7ra/AMQAr4mIAinAXe7uHYHFbj2T+sAbqrrUbZsHvC0idwHf\nAre4x0sRkbdxHiIoAO4L1hNk+4/l8vjSzbxjxb+MCY4PZ8LeDVV7zE4D4Lp5Z21u06YN8fHxfPjh\nh4wZM4ZFixZxyy230KRJExYvXkzLli05cOAAw4cP5+abbz7r98Hzzz9P06ZN2bRpE+vXry811f5v\nfvMb2rRpQ2FhIaNGjWL9+vU88MADPPnkkyxfvpx27Uqn3JOSknj11VdZvXo1qsqwYcO4/PLLad26\nNVu2bOHNN9/kpZde4pZbbuGf//wnd9xxR9X8Xbk8fdFSVT8APiiz7gW/z18Cvc+wXwYw6CzHPEjp\nR5n9234D/OY8unxe8gqKeO2L7cz/ZAunCgqZfnlP7r/yApo3svdZjakLitNkxQHmlVdeQVWZNWsW\nK1asoF69euzatYt9+/bRqVOnMx5jxYoVPPDAAwAMHDiQgQMHlrS9/fbbLFiwgIKCAvbs2UNqamqp\n9rI+//xzxo4dWzKb87hx41i5ciU333wz0dHRDB48GCi/HMD5sG++KvIft/hXRtYJruzbgV/eGEt0\nu2YV72iMqXrlXGl4acyYMfzkJz/hq6++Iicnh6FDh/KXv/yFrKwskpKSaNCgAVFRUWecnr8i27Zt\n44knnmDt2rW0bt2ayZMnV+o4xYqn+Qdnqn8vUmR2l/k8fXvwBHe/lsidC9egCgsn+1g4+UILLsbU\nQc2bN2fkyJFMnTq15Ob+0aNH6dChAw0aNGD58uV8++235R5jxIgRvPHGGwBs3LiR9evXA840/82a\nNaNVq1bs27ePDz/8sGSfFi1acOzYse8c67LLLuPdd98lJyeHEydOsHjxYi677LKqGm6F7AqmknLy\nCnhueTovrdhGgzBh5nV9mXKJFf8ypq6bOHEiY8eOLXmi7Pbbb+emm25iwIAB+Hw++vbtW+7+9957\nL1OmTCEmJoaYmBiGDh0KOJUphwwZQt++fenWrVupaf6nTZvG6NGj6dKlC8uXLy9ZHxcXx+TJk4mP\ndx7CvfvuuxkyZIgn6bAzsen6KzFd/7qdR/jR35LYm53LuCFdefi6vnRs2diDHhpjAmXT9XvDpuuv\nZt3bNqVXx+Y8d/sQhna34l/GGHMmFmAqIbxpQ/52V9lZb4wxxvizm/zGGGM8YQHGGFNr1OV7yl44\n379PCzDGmFqhcePGHDx40IJMFVFVDh48SOPGlX+Aye7BGGNqhYiICDIzM8nKygp2V2qNxo0bExER\nUen9LcAYY2qFBg0aEB0dHexuGD+WIjPGGOMJCzDGGGM8YQHGGGOMJ+r0VDEikoVTU6ay2gEHqqg7\noaCujRdszHWFjfncdFfVCmvO1+kAc75EJDGQ+Xhqi7o2XrAx1xU2Zm9YiswYY4wnLMAYY4zxhAWY\n87Mg2B2oZnVtvGBjritszB6wezDGGGM8YVcwxhhjPGEBpgIiMlpENotIuojMPEO7iMjTbvt6EYkL\nRj+rUgBjvt0d6wYR+UJEBgWjn1WpojH7bXehiBSIyPjq7J8XAhmziFwhIskikiIi/6nuPla1AP7b\nbiUiS0RknTvmKcHoZ1URkYUisl9ENp6l3dvvL1W1P2f5A4QBW4EeQENgHRBbZpvrgQ8BAYYDq4Pd\n72oY88VAa/fzdXVhzH7bfQp8AIwPdr+r4d85HEgFIt3lDsHudzWMeRbwe/dze+AQ0DDYfT+PMY8A\n4oCNZ2n39PvLrmDKFw+kq2qGquYBi4AxZbYZA/xVHauAcBHpXN0drUIVjllVv1DVw+7iKqDy063W\nDIH8OwPMAP4J7K/OznkkkDHfBvyfqu4AUNVQH3cgY1aghYgI0BwnwBRUbzerjqquwBnD2Xj6/WUB\npnxdgZ1+y5nuunPdJpSc63juwvkNKJRVOGYR6QqMBZ6vxn55KZB/595AaxH5TESSRGRStfXOG4GM\n+VkgBtgNbAAeVNWi6uleUHj6/WXT9ZtKE5GROAHm0mD3pRo8BTysqkXOL7d1Qn1gKDAKaAJ8KSKr\nVPWb4HbLU9cCycCVQE/gIxFZqarZwe1WaLIAU75dQDe/5Qh33bluE0oCGo+IDAReBq5T1YPV1Dev\nBDJmH7DIDS7tgOtFpEBV362eLla5QMacCRxU1RPACRFZAQwCQjXABDLmKcA8dW5QpIvINqAvsKZ6\nuljtPP3+shRZ+dYCvUQkWkQaAhOAhDLbJACT3KcxhgNHVXVPdXe0ClU4ZhGJBP4P+GEt+W22wjGr\narSqRqlqFPAO8OMQDi4Q2H/b/wIuFZH6ItIUGAZsquZ+VqVAxrwD54oNEekI9AEyqrWX1cvT7y+7\ngimHqhaIyP3AMpwnUBaqaoqITHfbX8B5ouh6IB3IwfkNKGQFOOb/BdoCf3Z/oy/QEJ4oMMAx1yqB\njFlVN4nIUmA9UAS8rKpnfNw1FAT47/wY8BcR2YDzZNXDqhqysyyLyJvAFUA7EckEZgMNoHq+v+xN\nfmOMMZ6wFJkxxhhPWIAxxhjjCQswxhhjPGEBxhhjjCcswBhjjPGEBRhjPCAihe4sxMV/zjpDcyWO\nHXW22XGNqUnsPRhjvHFSVQcHuxPGBJNdwRhTjURku4g87tbSWSMiF7jro0TkU7cmxyfubAmISEcR\nWezWJ1knIhe7hwoTkZfcmiX/FpEm7vYPiEiqe5xFQRqmMYAFGGO80qRMiuxWv7ajqjoAZ+bep9x1\nzwCvqepA4HXgaXf908B/VHUQTl2PFHd9L+A5Ve0HHAG+766fCQxxjzPdq8EZEwh7k98YD4jIcVVt\nfob124ErVTVDRBoAe1W1rYgcADqrar67fo+qthORLCBCVU/5HSMK+EhVe7nLDwMNVPXX7tQux4F3\ngXdV9bjHQzXmrOwKxpjqp2f5fC5O+X0u5PT91BuA53CudtaKiN1nNUFjAcaY6ner388v3c9f4Mzu\nC3A7sNL9/AlwL4CIhIlIq7MdVETqAd1UdTnwMNAKpyqjMUFhv90Y440mIpLst7xUVYsfVW4tIutx\nrkImuutmAK+KyM+BLE7PavsgsEBE7sK5UrkXONt06mHA390gJMDTqnqkykZkzDmyezDGVCP3Howv\nlKeANyZQliIzxhjjCbuCMcYY4wm7gjHGGOMJCzDGGGM8YQHGGGOMJyzAGGOM8YQFGGOMMZ6wAGOM\nMcYT/x9iLAkO3Wk5eQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdb308c6eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.figure()\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.plot(hist.history['loss'])\n",
    "plt.plot(hist.history['val_loss'])\n",
    "plt.legend(['Training', 'Validation'])\n",
    "\n",
    "plt.figure()\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.plot(hist.history['acc'])\n",
    "plt.plot(hist.history['val_acc'])\n",
    "plt.legend(['Training', 'Validation'], loc='lower right')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Step 4: Evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available Metrics in Model: ['loss', 'acc']\n"
     ]
    }
   ],
   "source": [
    "print('Available Metrics in Model: {}'.format(model.metrics_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Loss: 0.0933376350194\n",
      "Test Accuracy: 0.9685\n"
     ]
    }
   ],
   "source": [
    "# Evaluating the model on the test data    \n",
    "loss, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test Loss:', loss)\n",
    "print('Test Accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Let's plot our model Predictions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6IAAABYCAYAAADvGUplAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcTfX/wPHXnX2zjd1YZhhjLFkaREqoSCEqla8l0Upp\n1aqkVZsQKpX6tYgiJCUlosigEDPWmbHvWSbGmLlzf3+875lxxwyz3eXc+34+Hh5z59xz7nw+zrln\n+bw/n/fHYrPZUEoppZRSSimlXMXP3QVQSimllFJKKeVb9EFUKaWUUkoppZRLOf1BNDQ09IDFYrEV\n919oaOgBZ5fNGbS+3l1nX6sv6DGt9dX6mrnOvlZf0GNa66v19cX6mrXOvlbf/CzOHiNqsVhsJfkb\nFosFm81mcUKRnErrW6xtTVdnX6sv6DFdjO20vibga99hX6sv6DFdjO20viag9S3Wtqars6/VNz/t\nmquUUkoppZRSyqX0QVQppZRSSimllEu55UF04cKFNGrUiNjYWMaOHeuOIrjUkCFDqFatGs2aNXN3\nUVzG1/ax1te7+Vp9wffq7Ev11WuSd+9f8L366jHt/ftY6+ulbDabU//Jn8iTnZ1tq1+/vm3Hjh22\nzMxMW/PmzW2bNm2y5Wffzunlc3Z9bTab7bfffrOtXbvW1rRp0/Pe88b6evM+9rX6FlRnra9317eo\ndfa1+pq1ziW9Jpm1vgXV2Zv3r9ZX6H2WnqPNWmdfq2/+fy6PiCYmJhIbG0v9+vUJCgri9ttvZ968\nea4uhkt17NiRyMhIdxfDZXxtH2t9tb7extfq7Gv11WuSd+9fX6sv6DHt7ftY6+u99XX5g+jevXup\nU6dO7u+1a9dm7969ri6GciJf28daX62vt/G1OvtafX2Nr+1fX6uvL/K1faz19d76arIipZRSSiml\nlFIu5fIH0aioKHbv3p37+549e4iKinJ1MZQT+do+1vpqfb2Nr9XZ1+rra3xt//pafX2Rr+1jra8X\n19fVg3CzsrJsMTExtpSUlNwBuBs3bvSaAbj562tITU31mUH03ryPfa2+BdVZ6+vd9S1qnX2tvmat\nc0mvSWatb0F19ub9q/XNo/dZeo42Y519rb75/wU45en2AgICApg0aRLdunXDarUyZMgQmjZtWuC6\n1/r1tbm4eEX2c843lqKu269fP5YuXcqRI0eoXbs2Y8aMYejQoeet5y319YZ97Gv1haLXWevr3fWF\notfZ1+oLnltnZ1yTwDvq6w37F/ScdSF6n6XnaIOn1tnX6gtFq7PF/jTuNBaLxVaSv2GxWLjGcosT\nSlQ2CvvP1foWnSfX2dfqC3pMG7S+ReNr9bVv67F11vrm0WO6aLS+nkXrK3ztnOWt9YWiPYhqsiKl\nlFJKKaWUUi6lD6JKKaWUUkoppVzK5WNEVfGlvdweAGuIhO6rNj0MwMoWs3PXafDrnQCUSwwFoPrE\nFa4solJKKaWUKfmFhQGQsCIdgNFV1wHQNekmgq7d6bZyKeXtNCKqlFJKKaWUUsqlNCLqwY4taAjA\nxpaTCnw/65yxzZs7fwTAl61rAvD1z1cBYE3e5sQSup8lQbKILfjucwAuef8BAOq8ZO6IsH/FCgBs\nmVQ/d9+OOpQAwD/94wCwJm11T+GUUqqYAmpUB+Bsw1rnvRe4dS8AW56uD0DFJMlvEZl8BgC/5X+7\nooiqAGd6tgUg9Me/ALC1bkJqr3AAruzyDwDLf73EYZuaK60AhMxPdFUxS8yIhG6d2giAuVWnApBj\nf3/3+po0QCOiyntsf6cdO257H4BBOzsCcLD9SbeVx+kPoiEhIQctFkv14m5XPrgCnHVGiZxL61t0\nZqyzr9UX9JguKq2vOfjad9jX6gt6TBeV1tcctL5FZ8Y6+1p983P6g2hGRkYN43Wx5rox6X9sWdX3\n2IKG/NFyRoGrvn9cWo3HrbyW6HoyXnRRk28B6F9uPwCvDK4CQP0nnRsRLXF9oUz28aE25QHIRlpg\nw/Y5dzqic+sLzjumc2JqA/BPpw9yI98vV1sLQIs+lwNQx0URUWd8h62dLwXggalfA/Bew9hilyv9\ntnZUXHdEPm/L9mJvXxh3n7OOD5Ix4avGvgdAk8nDAKj7eiK27Oyy+SPnKMv6BtSrA0C1mccB+G1t\nEwDip8jv1k1bSlRG/6pVATjaXY6TSjPt0ZnMzGJ/lrvPWa7mrvqeGNAOgKPXS1TzqVYLARhU/ofz\n1v34RF0Abio3B4BKfUMc3u8RlVCsv+3u77CrlWV9/atUBsA6U/JNzGg4DoCD1kAAKvgtpW5AmONG\ndyxz+PXQgNMA7JsYxL2vPgRA5Q9XFrlYF1OW9U15tgUASZ0nAtA/pTsAR1+JAaDBwj9LXM6yosdz\nMRSxzkYPjRMdotl7rfyJ1F4SDc+yyf1kh3W3A3B4dyUAmow9QHbarmIVpyhcdV9p6NAuKff1Z/Xk\nu3tln3sBCJuzqvgfWEo6RlQppZRSSimllEvpGFEPk321tPz+2mIyIC2Q44/JmMAlt7WWlfYdAiDu\n2Br8QqTl+NVVMkbjmSoyZiO7UtlHTjzRsebScrUnWyIjlT8uu1ZXdwioI5HQmKllF+HzRDu7BQMQ\n6f9fiT/jwA1nyRoobWmRPcqkWG4VECVj5156/iOH5UnDpwDQfeKV2NLTXV6uogqoUZ0Xl0om70aB\nMsKqy1Fp6LVuKlnPDCMS2v93iYC2C5GI2fB/pPWWvzeVuLzOYESTtrwjEb5ODbex96osoGTRW7Pw\na9EYgM0PhrO863gAqvqvlveK0N49tIIRZQi54HrK+bZOkGN3S/zH9iUS/azmL79NOR7HX+myzp5T\nFR229bfI935Bo/m528wc9SYA9yVL/ga/39c5rewlcbaa473ShuWSmyNmobnvJZQjS7Dcc6SMkd5Y\nk26R6+xVoadz18myybkqxz5CeHnL6fJGS/uPykOo29cVpXUuIwp6rn0dZVx+7BxXl0YjokoppZRS\nSimlXMxjI6JH75ZxUnUHSmRo86HqnM2UCGHUV/IzbI9EU3LWJRXwCeb0X1QQIK3IRiR0aS+JdlpT\nzh9ftX1MKwCmR75tXyKtPrUXencbg62DNFEt7yHjV65a9iAAsZgzu+Ku52XcZ8J1ciy/UXN5oetG\nXC7jgnc/J9tU2SAtuqHzPD9DoSVQju8uXUrfKl7u7xBuHfobAEsqSiTZevxEqT/XXQ51qwdA17As\nh+WXrrkNgKr/eWaW5IDaUQBUmHma5kESNmn0y30ANLzjr1J9dvLL0QDcGiHjCy8d/wQAtf72rKzY\nhx6Q7+Lohz4D4IawRbnv9a7SE4DsvftcXzAXORVTDoCt3d8DQou8nZHv4MudbS64XgU8t4eIX8sm\nnKkhWWTTektU4Za2Eg3Ossn3Ycnnknm25m8nsHlYFP9ctvYtmHn5B/bf5PZwYYZERMeOvAOAcpuO\nwOF/AfA7tttxez+pb9zbMq496dZ3aRAYAUDGKMnKWWGwjM3LPnDQSbUonsAIGWSXniM/6/7svT0X\nLsTaSSKFAc8fZH6j7wAItMj+zD9msvKzcg9uSdvL0Z6SByBy7kYAcjy0186ukfZZBwZOKHSdO3de\nDcDH9X4u8P11l0+jFxc+V5lV7CPuGwvt3U8rSimllFJKKaU8jsdGRJ8YKX2zbw4/JgsanPNmJ/mR\nli19uycc7lyiv5F4SCIQ4W/LnI0Bi9eW6HPKUsXPZFzCLWsGYDkmLYjZ+9MKXf+u638BIMIv2NlF\n8yj/NpFW95r+0lobNSvQncUptQ33vgvktTxeyNIWX8oLSfbHnFMyd+y09N4E/Or+Y/hC0vtIq+vE\nKKlv47kybqghxc/UllnJxohKmwFYWk7GqGHCiKgxj123Eb8X+H7wDMnYh825GaFL6lgHyZQ7N3py\n7rLGo2Qce2lGqtvat2B7D4nOXPWPDMypM03298W/Ja7hHycXpo8ek3GRLYPkkppzzjr735NoYc17\nZbxs9v4DritgGQqoHUXyk9LzoPoKifyV/0pa0f0y5djcmnWW3dkybrBOgGRKHrxRImnHkmX8bPXV\nsm7FFbux/Se9mioc99yIZ35Gb5yU4fL79PYfkmDvCVCokdJbJePxs0w9LhGkKetlru+GQ5MByDlz\nxgmlLZ6sCkHnHMOyn0Z+MgSAOnOkF8IFv3s58q4RXWkc9AAbbpQI1G+XzAKgwzUSLa3whfsjov6x\nMWzqOA2Ah/ZJNMx/Sel6cZiFMWYyvZccz6Nfk/+Hq0JP556/jIz9+cdMXvrcYABa1PBjXrTMc9+m\novRKq/6uZ/VWsbWXG6VpQ94tdJ3mn4wAIOYl2ffx78iXe/ONkwvdRpUdj30QnfiMdAF4vrkEbSsl\n2zjWWC5+Qc3lAvdGM5my5J2achO74HQEN4QVnPwkwybdLlZlSheaTiFZYN8u9jZJfBG3uMyrUWLW\nIkzNkfZKe4ZWfMv+myR5eGy/pMwv94tc3Dzlhq2sXT1MHtjn2pMlRCyVbstmq2/gUnmINLrAXMjf\nZ+VikJYlCVz6hEv3qFsj5Kb/1s+nFnuaA1eydWjJ5NflpuSLk9IIFD9KjvOS7Lf2XTeWVdHcKvNy\neYh+udrHDstP27uKlZ/u/ukDCmJM1XL4xrwb6NZvyc1Ijd0lvxkxbhxGffl/ucv+WyAPceFHU0r8\nuc6Q/JQ0EjS/wIPIqgS5edu6UvbnTZ8/CkD9V2QYgSc8gFyIf0VpqG27IJW5VaTLXoc1DzisE/yj\ndEUdecPg3Cl6/BtL0pfILTvkZ47jNc0s6fRyrpAb9TR5fmJBB7k5bRBgdEH25+cMef1MUm8Aju+S\n69LG3nLz+9xBuS6/UWMNLUJ3AjCu7UwAnn5kMAC1X3P/Dbw1xJL7uvmKwQDUfaXk5Wo4fBXfXyPX\nuL4RRwE43usUABW+KPHHlpktL1S8+EpFkNldumum18m7pa66VgIJtrWe2RU7s5MM+fp1/CSH5Usy\nInj+ZWl8CDzt2Ph5sp7cjwfZ8/s88fg0TuTINzliv+fdfdnat8D2stwnJdhjNcZD9pz/qgEwbXAv\noldJQ5HN3pDS6JH1AHSfez8AL70v07q0DrZyzUbpevxLs3JOL7+zNJh5Hztue99h2fZ35Bzlji66\n2jVXKaWUUkoppZRLeWxENHzWKvvPvGXl863zbo1OALzcIVre/207b3SKLfDzAjKkHSR8w34AKi+b\nzSVB9qRHaebq1nl8oCRy+mPQW1Twk0joykxpkV/3siQvCj3p+YlrSsK/aSMAXq32FQAfnzRvkpqM\n3m25s+Y3QF6X3IK65jZbLIlfqi6WJr3gE7LO052kHemfvhNz193ztCRN8YTW9fyOPX2a2gHSevro\ngzcAEHis+F2JA2pKdOyTugtz062bWepNBUfTbtnW2/7KM5Pc7J4gSUi2tf0UgFGHWhL1ibT+l6Zt\nfG8n6bXSITiHZiukW2ddD+vuBeDfJI5frh5v/00iYq8flej2muMyvcXMBgtz14+zJ+r6sP97su60\nGwHISd3piuIWmzE1WOYsiYg+U+VXGn0rYcH4OQXvZyMaCmBNLtmUPZ4kZXpLvmz/IcA53W9lX/dL\nvRaA1ZtjiH9IeiBVPSX1r2pf876EawA4NEJ6gDzynj+jqi8FYHmGRArXPSBR095fyPGQvXuPcypT\nBI2ezove+a8tm4jPs6vlPNa3s/T4GN5Upo74nkpl8vml8c5lM3Nf/zFdho3U4OLnmh1fyn3WhMvk\nPuSSIBlWUd0/b4jU9iy51t046xEAGjzuGT1bjB4nr733gcPyfjuuB+Dk6DpUWlLw1DUVYmMAaPmN\n9HJoHORH/DypX9ys4g+vcbZDbcJZHS9djo1eZyfsPY1Gfy29LqNXnl9XY6qtwEVrABjwk9yDbeo5\niZGRUvcPv5JrU0y/9c4qvtPkj4a6m/nv4pRSSimllFJKmYrHRkSLwkj/HT5bflqB8FlHL7jNwbsk\nmtg0KIC3/pXoWvQnMu7ILGNWjlwq/faNaCjAHUvvAiBurndGQg17r63s8Pva9Hr2VxmuL0wJGVHd\nl8dNpXXQWWOpwzpGAqJRS26m8RP2BC0nTzqs02ibTO+T2EuOg7bBZ/jx/jcA6Boi01xEvyoRR6OF\nzx2MqZi+ueRNPjvRHIDAX0qeVCnpRRmbmGWzckeaRByshw6XspTuc0MbxxbVEzlyLGe9INMc+Hlo\nRNRmk/FkRhR/1dFo/DMOFftz/MpJ5GXLK5LEZW4vmZIph0Dq9v2nLIrqFEfaViY6QBJN3bO7IwB7\n2kmOAr9wGUSVcN+DPH731wD0Lyf/Nx3tp+35s3cBkHSDZyUx8q8kkarNL8n5ZUvjKQCszYT4F+Va\nmf9c5C38wiUav+1FGT+XfNVk/Ozn5tX2hEz950kik0ZjJAoad3yNQ3Kqc11Sbi8APwdIJGnNmwlU\nHieRo97hx+1rWQra1KX8mscD0Kniz2zNkjHLVTZkXWiTIqv0m/2AL1lOSafwLy/968L9MlmUIfu8\nxjsFR0ItgUGc7SzXrWff+wSAjiFy/TKibImZEgkdtLkvj8bI1E297OeAKb0lEjx+Wh+gaPk/nOnY\ns3J9McZMXr/5JgD8H5f/E/+/C0/WdDxBrkmjq32du6zOosLWdj+/a47mJloyEi/dmdILgOjnCo76\nFiTufrmvfveKpjwaKfdj/ZvIuPgVBJVVcX2WRkSVUkoppZRSSrmUqSOixWFkeJz0jGQIC7T4880E\niaZU3l/0lhF3OvuzRP9Wxr9tXxJCi5XST73xY9Jv3fPylpWtk00cW2nXTZKMhhUxxz4EyLGnx8+L\nhuYZsvM6ANJvk3FIcXsSC92nRsvqsE9l/MKae8dT01+2+2uojF27+Vs5Pmzrk8um8CXg1/sIALUC\ngvl4utSvdhHG4eRnRJK/uFrGtmTastg1TqI24ZmeNz6lKDKvb8OkqA8dlu2xd83w++1vN5So5H6I\nn8vQpRL22JUuEf2zH9codP0DV0oT9fWXrQPgu1pT7O/ImP0O626nEp47ztAanDfFxYYPJIIWaT8P\n5ZySzKA1317B1z0lo2a/ct/LhjZpoT+YKZFg2xn39VYoyL4BMs51Sx8Zu/jdKYmQftzjWqyHd7it\nXK5wvJc9k2hfyUbvRxiLMyR0NHaYnEtjF8lYv4LOy5YAObf7NbJP6zM3EoA3P5MM0JcEHQIkiu5v\nkTjAJav+B0DUIff93267Q7LH3h5xmCs2DASg/A+r3VYeZ0t9uBkAV4QspsmSQQDE4ni+9bePh9wy\nvDpJtzpO/bE4Q8bHD/tpMADxE+QaF7x1B5ORa9K7i+We8/t4md3htboy1jooqUyrUiypM5qzqZVE\ndfdkS2TU71n5ftv+3lDodsZUL7EPS+H97DGsO3deTagH9sILiKoFwGONfjnvvZRvJJt3dYrfi2ra\nvGt49M7NpSucOo9GRJVSSimllFJKuZTPREQ3PxIFQJtgGY+x6WwGkUmn3VmkIguoHw3AS7GSYbWS\nfWzo2kyo95K0y1qPHXNL2Vwps3sb5nWVlskXj8h8mZGzpRWvsDE6ZvHMwdYAnLxLxsBa9xQ9EhQ9\nW1pjn+vdjrE1PKcV27+q5I4cFbcgd1ntV0ue/XTzMGm1bx0sx/zkY00In23OSKjhYJvzM3b3/P5h\nABri2XWr9q5E35dMlfNR59AzfFx3CQB+9nFvOeNsBW987jo4rvNVuoxDqvxMgEd/r8vdvD/39Ylu\nEgGN/OT89Z6v9539lWO77/K/ZVxe3DHPiiikX+Y43n5C6tUAhG717mgogM0+VP+MLW/cZnqOHOcH\nLpOxYBk3tQUgtuE5+/+MfAf61pPxdcMrfg7AmrOyTYdg40gOy93mjzOyLOpl+VvuHMf/SHc5R2/N\nOkPQZCMPg/fub0vzvDHOgTtCC1zHmGN0c+fJueeh/indATj5hNxPNlwp5+iCouPbU+y9QeJLX96y\nMqhJYu6YyZ3Z9nko/rx4JHTLeMm0O6+uzKFr/H/sfLMRYR54nTp2hWQtvyViXu6ye3Z3AiDKnvG3\ntDlhmoVKduvE+l3k81LSSvmJvsvrH0Qzb5BuUX/d8o59iXyx7n/oIUJXeNYNQGEafC0JD1oFOd7I\n9Ft8H3HrPefBw9n2dAmgeZBc8O9Iky5U1U6Zt5uEkegAYMOlxs14CboiWuRGJsAvx+EzAfaNkZ81\neuffyPksYbKvuoXJ1DptVw+iBiXvIlwl+l+H379MbU0V3Jv4obSCWuU1ICWflYax+InSsODp3ewD\nfpWEHROukAvxS5dHs6erHMfbe0p6+MRMOTYHLLrvvO0bfiY33gu+meaw/I2kbgBErffMieAN6bNr\nQlN5PbiJ3IwtayMPKYdbSdc9W49/aRYo15nkLBlW0NQ+jcuc7tKo9mS7u+VDLnBD6EpfdZhqfyXX\nm1lNvgCg/bjHiPlOhhP4Ly08oYmZVZonx9w9g/oD8EX8F/QKl2P65vul67jV5tg8kmnLJtiS/1ZK\nfs97ABXZWOm0QaaNiBwu33Bbiucc5x8c7UjI9+a4LyqN+GoHC33PkiBf6jlXvGdfEkjTpfcA0HCo\nXL8sZ4o+Zcfzh+QeNGSpJF7z5MY1gzEMJvlB6U68uedkh/eX2Lsml1uR6pHXqcOXnp8AbMdYGXIQ\neqBsju8e4ZIYdVxraXCIMPmDaOwj7pteSLvmKqWUUkoppZRyKa+PiO7qLs/aERaJhBqTUIctXE/h\nncY8w7E7ZNqLMdWN5ERSB2PKisZPbPfI1ihnqdrsUG5rdMA890+GXVJb7pfuWca0F6WVdpN0pZpV\nNZEse98y47NrjZZ13NEKm/OvTE/w0mGZKPx/DdawrKYk8SjOVBVGorE/Ws6wL5HvdMafVcCkEdEz\nPSRytqbNexhT92zJqgaA1WRdII1ptMK+PUic5OXg+vsudVgnjvNboY0pI4wuui8fkQQi9R6SCLqn\nT6dV47tUtj4tEcKRlSWJx5NzJWJybnfj23bcAEDGCOmq3uerpQDcWX43ADtGyPHcwDPmu6dtsHQX\nN84hxlCQzbdNJutWWdZssUS4K6yW9/6rLfUtnwJVNpxy+LwjzWV6jOpLZfoaTz6+c9LTAQjuKj/v\nqX4TyS9EA9A1QSJaW0/I93Tn3ioA+AdZ6dVIotlv1Fhzwc9vsuQeGj0mPZyyDxZ/qqOy5l9RIl7l\n/Pa4uSSuVTtMrk1++IHF8U5w6wi5z2ocKN+DhNUDaNBfEhkV5zoaGCHnhlPZ8nk5Z86UpshlYnZq\nS0ZWluO4VbB8T6/ccH652obJibxzqLyXv96Prb8FgNoHPSeafy5rmJTY75xYW1klVTJ6nWV5+gOE\niWhEVCmllFJKKaWUS3ltRNSYJH3glb8DcDJHWnYOvVofgOBMzx5bGRBViytHyLijCL9gh/dWJsUC\nEHfMs+tQVgJiZNqatxp9w4cnJDoWOc0807XkN+rK+aXaPqBObQDSEyRF+ft3TjlvncRMiVRYzrov\nrmREFxbtlcjX8pbT2f+9tMAv/6B9odsdbyJNjRHREhlrVytNPi9fu6zFxC2SGVWkVfXcMb1PrJWJ\nxWPwjLGCzrZrtNTdiB4ueqUjABG7PSQ0eBHZ+w9wz0hJLPXJW+MAiAuU6J8xRUvsoruJf0DGseec\nkqjp2F97AjC0t4xBe721RB8+anEDOW6cZskQM1/GrG7t8f557xnH65Zr7FMOXVP0z018SiLfDyfZ\nx0j28PzeDNaDh4i7XyKXafZlQewEoKH9J8CiOU2A8yOiadky7rv3u0/INuMTsWZ7Tqx/z1AZD9m/\nnCQZ++tUdJn/jczrTzj8fjonqMz/RnHl2CQGk0MO2BzHE9asfjzvPaBJ1YMUJxWkMe3Lpo4y9r3j\nhlsBKO8ByZ9qDNhLr7l9APg+XhL5GBHSglz55IMA5PST8ZDLW04HoNqHYYVu4wmaN08Dzr9nKAtG\nTxFnfLav0oioUkoppZRSSimX8tqI6LYXpKXv+yoSLbpx280ABJtkkubkZ+owt4Zj5KzzP30BGRsK\nnp9Vs6xsu1cif+2C4e6/OgNQh43uLJJbJY2RLG2buk46773Z/8m4pfcel2MlJNn9GRArjZHo7FUv\n9GNOs08BeH104RHtNZkSdbHa28laB521v+PYcl333X9M2yaZ2ft47msjW27tj86fysUbHblHouEb\n2kkmxjT7xOqhh88Wuo2nivhGeq3cyaMA/Hur7MszJ+zjzEbuwHrKccxko6ckMnp1Q4mA/9x0NgCj\nR/sRdZPzy3wxjYbLeLhu30im0EGT5DoU5pdJjzCZBD5/du6iaBsske/fW30JQNM3R9BgpHl7thhS\nX23PX22MrPyO0b5b3pBIaK3JMm2ViTtxlEh2lwRmtDKuU/KdmPO6TAdUAc/s+VBxqPSeW7VczseT\n6s6n/euPAxA3UaLg2Xv3Fbp945myzkGrnNdCJkTa33F/RDQnPR2ulp5KXfoMA+BQgmM8qlKyjQpf\nyr45/LlkNt9sz8/w8YloAMI2ydRFnhPbd72d2XK9MuN1y9NoRFQppZRSSimllEt5XUT0xIB2AGy4\nbSIAO7Jl7rb/XpdxdcHsL3hDD7O21zsYLYiGCsMk/pN9rDgjFswvp05eVreM4yFuLIl7BS6tCcBr\nNWcXus6ney8HIGS++yOhuRJlDEqF62FgpxEAHG8YXOjqlT90jJLs/VZ6N6y97FOH5cYYVDPxj5Os\nwWvafGEs4cf/JFts4C9r3VQq1zp97X8Ov9+y7i4Aqi0x79yURmQ04hvH5QX1WjGO25NzZL8bc5G+\n3nw2U2p2AoqXVbqs2exjGI3j8av4WrnvTbxFxndaA6V3wuWPy3lmbI2i9zQyMlnWbmGOa3Fh9o2U\nc+1P/d8g1OI4Zm7CMcnjUOOTdYA55o4sS9ldEgD496FTxAfKuX7Y3g4AVJwp33N3RIeN8ZsdK/xa\n6DpGtPP1a2Ty7RazU9g4QO4nh10lPbL23yBRTutRmdv6+EDp5XHFw6t4vvofACTMkChqg4WeGfkN\nmyPnrOg5ha+zuctHQN54yMlbrgKg1u4k5xbOQ93Ve1Hu6xs/GQlA3SUr3FWcEhu0syOf1VvmsGz7\nO/Ls5I7TnEnsAAAL6UlEQVT5RDUiqpRSSimllFLKpbwqIhoQVYuHn5sJQLBFqnb7+oEAVP3RHGND\nLySrumQcDTwbVeg61sNHALBlSt9+S7C0RvpXrZK3TtWKAGx7rODsdTarhfgH7eNQT54sZalLb8pl\nX+S+jvqx+GOTPI2/RVoXzx1ndfJ/7RzWGfPix0DePF7nrp83/+j5/xe2LnvLsqhlzn+ptIZXXlr0\nbTLSJAM2lzkut3VoieWPdWVTMBc52FnmIDx3309aInMbN2SVW8rkah8kfA7AfquMp6w83rMzMDpL\n1Q8kmnhZ9/8BsCphOg89Hg1Ag8fcFxG9kPBZjsfo/BYSCRo7UK6vp21nSVh2PwD1PpJj/MgI2c95\nvQDMLatrawDmPvAGAHUD8o7fXfYsud89KeMgg0979n1H+TS5lhjZfUvLEiD3Xccfkaj/mktn8HNG\nKABbn5PQf1DWhedadSbr9lQAZhyQeZz7NFhIvSt2AeBfvrysY7/nyU5JA2BtKz86DpSePJEbZGy/\npYr0tEudJFn8N3WUcbAHrRl5kdDHPTMSWhT+TRvZX0mvCGM8ZPWJ5uiRdup56cWx5hN/WgfLMb7r\nm0sAqNu38CzBF9MmNJXETOkNEv3mesD3ejs4g1c8iBonvxbf76FvhKSZ/jJdbviqP2ek6Ta/BbOm\nXXSdy//uB8CRg3JSrVRVLgirEqYX6281GfUAAPWfcF8yiTM95WJxRYjRzdQrDlfGzpTJoG8dOj53\n2bI3JXFL3kMm9t/P3z7/OoZmi++jIebt3lgoe44iv3wdOMz2EApwJtIx4dLazLM0fl0mk/eFxA97\nnr6cDsFyjP6ZKTfw/ibuklsqOfI9rvy2/D8c+TyD5NvlPNBz+iAAbGs9c8J4Q92fpMETae8lzBJE\n8lXSiDawnjSw/BD9k31tx+/vrgORNMydFMU80nrIA3b0OQ+gRqPKoIcfAyBsgTkalcJnSzkXvtQY\ngAYhh9lWW7qMZ++5eKNmzhUtAUiVvDfc3FjOya9Wm5G7zquP3wFA6E+eM1zkzF1yfzRudnzuNCYP\nLZauw4nvS+NKxL68M/LhNnIH2WZECgBv15JpAY1r0lR7Ep9P3+pBAxNPLWdIGe0YpOj7twyfqGGS\nc7Xfb5Jwbfj4B1j95LsA/HyZTJc1uLM0KhTnupM6ozkAHULW5t5jR57y/Omn8jvdR1rzP6v3gZtL\n4ki75iqllFJKKaWUcinvCDG1kG4EL1X7PHfR5Fdl+oqK683ZOnVjUn8WN5tV7O1WtPqq0PdO26R7\nRZbNMT58/YbBAJxYl9d9N+p398dndvWScKDRzfrFI5cQMU+6ipg5DX79mdJ9OnFACG2Dz1xk7fMl\nZkr3mKkHJHHAsWEynUt86nbvnNLHvrO9YQLpavm6Tn93slVud3pf0L/fYnLsO3TomsEA1EO6SvlX\ntk9zUK0yANbkbS4vnzsYrfed/m8kSUMkIpr+ikz9UL6vdEv31MRcgWtkH7X7S6IEf16ad/35PPpn\n+ytp7860SXfGHkmS8Ch+xA5Tna+M4/Pvm4yeLHkJ1zr9Lj2IGswxRyS0MMMqpnLwe4kWrvm37kXX\nHxszFYCWQY63kmvPyp4dmDiUBr9uBjxrujnrVplKZdmNTam0QKZXeqfWcnnzxeUO6/rhV+i1p9nv\ndwIQ+6icwyP3mvN+81y29i347rIp9t/kXsOyuJL7ClQKNZf+S+suA4C8oQF7Okmd6i25+PanbpYI\n4teXSbKqlZnBRL5sju7JBYl5ItndRSiQRkSVUkoppZRSSrmUqSOi/k3iALhnxrzcZU2mDQcg+nPz\nDhQHCO2WStNXpZXVVsheKhf/b6FjP5sul5Y6267w3GX1Z9mnTUh0HKxdiW0OP93NSBrwZIcfHJZP\n/7Ej9bPN3+JoTZKxBc8/ehe7e0pL69buRe+zP2zafQDUecVIG+7d0/nkhDi2Rh+2ZrqpJCVnJA27\nsdZ6h+VHz0bkJhbzNTlWaQc99IBMg3HDXRKJmJsi0xRF3eSecrlL7NTdfN5Xejcsu0R6w1zXYggA\nfr975nhoI1Jb40GJmPSc1otnohcA0N6eJGT2f9LT5tkfbgPypgfwpAjZhfhXkro9vEqOzwiL49RT\nrx9tTMO75dpp1j4bn77VA4BDDy1jTFX7Oarq+gtsYZCbk2z73lwvna4YMFPG4cU8tdKj93N2Shpz\nO8mY2Il3ynQtp2Ikcv/TdRL57vbTw+d1wWr0kfRkil69QT7HFYV1kUNtwokJkKifEQkOOGPOPmg5\nGzYT9az0mJwzR3o0fDf4TQCuq/IoAA2H5/VisCRIQq2D7SUx6AePTQCgcZBcq+Ln30Pcn54z1rmo\nLjQ29Mrh9wIQO8d9z0waEVVKKaWUUkop5VKmjohuHmZvhQ3Lm2Kk9lJ7k5zNnC0454p55uLRvx4k\nFLwtG8q6OC6TY48QJZ2WFNzX7JV0+Q1f3eTRravFFTovkTh7ML9jP4nkBw4+CMDCpjINUdeNt5Pz\nqWSAttkTrkavOwyYJ6JQWl9c9z4AyWeldbbfp08AUBcTTSRtlb01NfkKAB6+PA2ApbtjicKzM6M6\nS3LHTwDI6Sjn6qbLJPoX+4KM2fKV49uQvXsPX/eRcd8Df5Hv/5GREnmp9rvbilUk2WkyBQZdYMQI\nSaGa3kbGucaPkvFzsTvN2UvpSK94ALqGyaAya75bix/GdCL8lLnHhkbaM72uXhbHuLlyzD1a6eI9\npOJ/k+9s0D+SQbj2a3JOjsE8PZesBw8BEDX2kMPyB5EsunGcPwWP+e8uC3emii03Ejr+3yYAVP7Q\nPPszP+umLQD833WdAfhgqtRtYY9xAHx9ZQIzpncB4KN7JMNuq2DHvg3XJclMB/HvpZu218O5Gsy8\nL7dnSpgHTBunEVGllFJKKaWUUi5lyoioMb/k4p5v25f45oTo3soYM7dFAqEEsRPw7ghJ+a/s0QJ7\n0sk+yDEeTgqQ4rCuN/8/FOTF1F4AnJoSBUDd2SaKhNrZsmUUUfRTEu1r/JpMvGhZV85tZXKHn569\niqSnZQzoylUSaYqfsA+ABgek5dp6pviZpL2FkSn4tpSuAMxv9REAQ9vZJ2r80/N7ulSfKN/P6vbf\nzT5+7ubHfwHAmi/bfOx8GasfN9v9EYWyYt2eyi/N5Jz0C5dedP36eObYZVVyA3rnpZOdNu8aAKJN\nFOEuTHZKGgDB/aoCcF+rhwAIfPIAax+UsaDx84c7bBPzrXzng5fIeTcn66wrilrmwuzZvLvNkXl/\nY/Gs3ikaEVVKKaWUUkop5VKmjIju6+APQN0Ax0jol+nVCDwpLRbe3IdfKZ9y9R4Awtnj5oKUnnV7\nKgB1+7q5IG4SMj+Rw/PltdEqa/aImTOc7iNXsFUrZJz8sUaS/bySZzVk+4QWoTL+1d8i7fZ/npE+\nKU3ekDGFevwqbzI7tSUjK/9z8RVNynpYcmwELpKfLIJetAEgjoIz4urzhHOZ8kE0v9eOyoDqld2i\nse333i+QUkop72c9chSAqXH1AajkBV3jzOrhL4cCsPnuKQAMmfYgAHVSzDdEQKmLsS2O5JnaMt1H\n9TW+NhBIuYN2zVVKKaWUUkop5VKmjIjWf0pah69/Kv9g+gOuL4xSSimlvFK90RL57DZaEn3UMdO0\nUUoVU/WJK9g4UV6HFtJVVamypBFRpZRSSimllFIuZbHZdBiuUkoppZRSSinX0YioUkoppZRSSimX\n0gdRpZRSSimllFIupQ+iSimllFJKKaVcSh9ElVJKKaWUUkq5lD6IKqWUUkoppZRyKX0QVUoppZRS\nSinlUvogqpRSSimllFLKpfRBVCmllFJKKaWUS+mDqFJKKaWUUkopl9IHUaWUUkoppZRSLqUPokop\npZRSSimlXEofRJVSSimllFJKuZQ+iCqllFJKKaWUcil9EFVKKaWUUkop5VL6IKqUUkoppZRSyqX0\nQVQppZRSSimllEvpg6hSSimllFJKKZfSB1GllFJKKaWUUi6lD6JKKaWUUkoppVxKH0SVUkoppZRS\nSrmUPogqpZRSSimllHIpfRBVSimllFJKKeVS+iCqlFJKKaWUUsql/h9ai55FX1WawwAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdb3cbcf2b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "slice = 15\n",
    "predicted = model.predict(X_test[:slice]).argmax(-1)\n",
    "\n",
    "plt.figure(figsize=(16,8))\n",
    "for i in range(slice):\n",
    "    plt.subplot(1, slice, i+1)\n",
    "    plt.imshow(X_test_orig[i], interpolation='nearest')\n",
    "    plt.text(0, 0, predicted[i], color='black', \n",
    "             bbox=dict(facecolor='white', alpha=1))\n",
    "    plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding more Dense Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(nb_filters, (nb_conv, nb_conv),\n",
    "                 padding='valid', input_shape=shape_ord))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 2s - loss: 0.1922 - acc: 0.9503 - val_loss: 0.0864 - val_acc: 0.9721\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.0902 - acc: 0.9705 - val_loss: 0.0898 - val_acc: 0.9676\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fdacc048cf8>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          epochs=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.0898462146357\n",
      "Test accuracy: 0.9676\n"
     ]
    }
   ],
   "source": [
    "#Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Conv2D(nb_filters, (nb_conv, nb_conv),\n",
    "                        padding='valid',\n",
    "                        input_shape=shape_ord))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.2394 - acc: 0.9330 - val_loss: 0.1882 - val_acc: 0.9355\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.1038 - acc: 0.9654 - val_loss: 0.0900 - val_acc: 0.9679\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fdacc064be0>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          epochs=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.0900323278204\n",
      "Test accuracy: 0.9679\n"
     ]
    }
   ],
   "source": [
    "#Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding more Convolution Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(nb_filters, (nb_conv, nb_conv),\n",
    "                 padding='valid', input_shape=shape_ord))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Conv2D(nb_filters, (nb_conv, nb_conv)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n",
    "model.add(Dropout(0.25))\n",
    "    \n",
    "model.add(Flatten())\n",
    "model.add(Dense(128))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 2s - loss: 0.3680 - acc: 0.8722 - val_loss: 0.1699 - val_acc: 0.9457\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 2s - loss: 0.1380 - acc: 0.9508 - val_loss: 0.0600 - val_acc: 0.9793\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fdb308ea978>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          epochs=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.0600312609494\n",
      "Test accuracy: 0.9793\n"
     ]
    }
   ],
   "source": [
    "#Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise\n",
    "\n",
    "The above code has been written as a function. \n",
    "\n",
    "Change some of the **hyperparameters** and see what happens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Function for constructing the convolution neural network\n",
    "# Feel free to add parameters, if you want\n",
    "\n",
    "def build_model():\n",
    "    \"\"\"\"\"\"\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(nb_filters, (nb_conv, nb_conv), \n",
    "                     padding='valid',\n",
    "                     input_shape=shape_ord))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(nb_filters, (nb_conv, nb_conv)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n",
    "    model.add(Dropout(0.25))\n",
    "    \n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(nb_classes))\n",
    "    model.add(Activation('softmax'))\n",
    "    \n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "    model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "              epochs=nb_epoch,verbose=1,\n",
    "              validation_data=(X_test, Y_test))\n",
    "          \n",
    "\n",
    "    #Evaluating the model on the test data    \n",
    "    score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "    print('Test score:', score)\n",
    "    print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 2s - loss: 0.3752 - acc: 0.8672 - val_loss: 0.1512 - val_acc: 0.9505\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 2s - loss: 0.1384 - acc: 0.9528 - val_loss: 0.0672 - val_acc: 0.9775\n",
      "Test score: 0.0671689324878\n",
      "Test accuracy: 0.9775\n",
      "5.98 s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "#Timing how long it takes to build the model and test it.\n",
    "%timeit -n1 -r1 build_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding Convolutional Layers Structure\n",
    "\n",
    "In this exercise we want to build a (_quite shallow_) network which contains two \n",
    "[Convolution, Convolution, MaxPooling] stages, and two Dense layers.\n",
    "\n",
    "To test a different optimizer, we will use [AdaDelta](http://keras.io/optimizers/), which is a bit more complex than the simple Vanilla SGD with momentum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.optimizers import Adadelta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_shape = shape_ord\n",
    "nb_classes = 10\n",
    "\n",
    "## [conv@32x3x3+relu]x2 --> MaxPool@2x2 --> DropOut@0.25 -->\n",
    "## [conv@64x3x3+relu]x2 --> MaxPool@2x2 --> DropOut@0.25 -->\n",
    "## Flatten--> FC@512+relu --> DropOut@0.5 --> FC@nb_classes+SoftMax\n",
    "## NOTE: each couple of Conv filters must have `border_mode=\"same\"` and `\"valid\"`, respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# %load solutions/sol31.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding layer shapes\n",
    "\n",
    "An important feature of Keras layers is that each of them has an `input_shape` attribute, which you can use to visualize the shape of the input tensor, and an `output_shape` attribute, for inspecting the shape of the output tensor.\n",
    "\n",
    "As we can see, the input shape of the first convolutional layer corresponds to the `input_shape` attribute (which must be specified by the user). \n",
    "\n",
    "In this case, it is a `28x28` image with three color channels. \n",
    "\n",
    "Since this convolutional layer has the `padding` set to `same`, its output width and height will remain the same, and the number of output channel will be equal to the number of filters learned by the layer, 16. \n",
    "\n",
    "The following convolutional layer, instead, have the default `padding`, and therefore reduce width and height by $(k-1)$, where $k$ is the size of the kernel. \n",
    "\n",
    "`MaxPooling` layers, instead, reduce width and height of the input tensor, but keep the same number of channels. \n",
    "\n",
    "`Activation` layers, of course, don't change the shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer 0 \t conv2d_12 \t\t (None, 28, 28, 1) \t (None, 28, 28, 32)\n",
      "Layer 1 \t activation_21 \t\t (None, 28, 28, 32) \t (None, 28, 28, 32)\n",
      "Layer 2 \t conv2d_13 \t\t (None, 28, 28, 32) \t (None, 26, 26, 32)\n",
      "Layer 3 \t activation_22 \t\t (None, 26, 26, 32) \t (None, 26, 26, 32)\n",
      "Layer 4 \t max_pooling2d_5 \t\t (None, 26, 26, 32) \t (None, 13, 13, 32)\n",
      "Layer 5 \t dropout_6 \t\t (None, 13, 13, 32) \t (None, 13, 13, 32)\n",
      "Layer 6 \t conv2d_14 \t\t (None, 13, 13, 32) \t (None, 13, 13, 64)\n",
      "Layer 7 \t activation_23 \t\t (None, 13, 13, 64) \t (None, 13, 13, 64)\n",
      "Layer 8 \t conv2d_15 \t\t (None, 13, 13, 64) \t (None, 11, 11, 64)\n",
      "Layer 9 \t activation_24 \t\t (None, 11, 11, 64) \t (None, 11, 11, 64)\n",
      "Layer 10 \t max_pooling2d_6 \t\t (None, 11, 11, 64) \t (None, 5, 5, 64)\n",
      "Layer 11 \t dropout_7 \t\t (None, 5, 5, 64) \t (None, 5, 5, 64)\n",
      "Layer 12 \t flatten_6 \t\t (None, 5, 5, 64) \t (None, 1600)\n",
      "Layer 13 \t dense_10 \t\t (None, 1600) \t (None, 512)\n",
      "Layer 14 \t activation_25 \t\t (None, 512) \t (None, 512)\n",
      "Layer 15 \t dropout_8 \t\t (None, 512) \t (None, 512)\n",
      "Layer 16 \t dense_11 \t\t (None, 512) \t (None, 10)\n",
      "Layer 17 \t activation_26 \t\t (None, 10) \t (None, 10)\n"
     ]
    }
   ],
   "source": [
    "for i, layer in enumerate(model.layers):\n",
    "    print (\"Layer\", i, \"\\t\", layer.name, \"\\t\\t\", layer.input_shape, \"\\t\", layer.output_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding weights shape\n",
    "\n",
    "In the same way, we can visualize the shape of the weights learned by each layer. \n",
    "\n",
    "In particular, Keras lets you inspect weights by using the `get_weights` method of a layer object. \n",
    "\n",
    "This will return a list with two elements, the first one being the **weight tensor** and the second one being the **bias vector**.\n",
    "\n",
    "In particular:\n",
    "\n",
    "- **MaxPooling layer** don't have any weight tensor, since they don't have learnable parameters. \n",
    "\n",
    "\n",
    "- **Convolutional layers**, instead, learn a $(n_o, n_i, k, k)$ weight tensor, where $k$ is the size of the kernel, $n_i$ is the number of channels of the input tensor, and $n_o$ is the number of filters to be learned. \n",
    "\n",
    "For each of the $n_o$ filters, a bias is also learned. \n",
    "\n",
    "\n",
    "- **Dense layers** learn a $(n_i, n_o)$ weight tensor, where $n_o$ is the output size and $n_i$ is the input size of the layer. Each of the $n_o$ neurons also has a bias."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer 0 \t conv2d_12 \t\t (3, 3, 1, 32) \t (32,)\n",
      "Layer 2 \t conv2d_13 \t\t (3, 3, 32, 32) \t (32,)\n",
      "Layer 6 \t conv2d_14 \t\t (3, 3, 32, 64) \t (64,)\n",
      "Layer 8 \t conv2d_15 \t\t (3, 3, 64, 64) \t (64,)\n",
      "Layer 13 \t dense_10 \t\t (1600, 512) \t (512,)\n",
      "Layer 16 \t dense_11 \t\t (512, 10) \t (10,)\n"
     ]
    }
   ],
   "source": [
    "for i, layer in enumerate(model.layers):\n",
    "    if len(layer.get_weights()) > 0:\n",
    "        W, b = layer.get_weights()\n",
    "        print(\"Layer\", i, \"\\t\", layer.name, \"\\t\\t\", W.shape, \"\\t\", b.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Batch Normalisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## How to BatchNorm in Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "\n",
    "BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001, center=True, scale=True, \n",
    "                   beta_initializer='zeros', gamma_initializer='ones', moving_mean_initializer='zeros',\n",
    "                   moving_variance_initializer='ones', beta_regularizer=None, gamma_regularizer=None,\n",
    "                   beta_constraint=None, gamma_constraint=None)\n",
    "```\n",
    "\n",
    "#### Arguments\n",
    "\n",
    "<ul>\n",
    "<li><strong>axis</strong>: Integer, the axis that should be normalized\n",
    "    (typically the features axis).\n",
    "    For instance, after a <code>Conv2D</code> layer with\n",
    "    <code>data_format=\"channels_first\"</code>,\n",
    "    set <code>axis=1</code> in <code>BatchNormalization</code>.</li>\n",
    "<li><strong>momentum</strong>: Momentum for the moving average.</li>\n",
    "<li><strong>epsilon</strong>: Small float added to variance to avoid dividing by zero.</li>\n",
    "<li><strong>center</strong>: If True, add offset of <code>beta</code> to normalized tensor.\n",
    "    If False, <code>beta</code> is ignored.</li>\n",
    "<li><strong>scale</strong>: If True, multiply by <code>gamma</code>.\n",
    "    If False, <code>gamma</code> is not used.\n",
    "    When the next layer is linear (also e.g. <code>nn.relu</code>),\n",
    "    this can be disabled since the scaling\n",
    "    will be done by the next layer.</li>\n",
    "<li><strong>beta_initializer</strong>: Initializer for the beta weight.</li>\n",
    "<li><strong>gamma_initializer</strong>: Initializer for the gamma weight.</li>\n",
    "<li><strong>moving_mean_initializer</strong>: Initializer for the moving mean.</li>\n",
    "<li><strong>moving_variance_initializer</strong>: Initializer for the moving variance.</li>\n",
    "<li><strong>beta_regularizer</strong>: Optional regularizer for the beta weight.</li>\n",
    "<li><strong>gamma_regularizer</strong>: Optional regularizer for the gamma weight.</li>\n",
    "<li><strong>beta_constraint</strong>: Optional constraint for the beta weight.</li>\n",
    "<li><strong>gamma_constraint</strong>: Optional constraint for the gamma weight.</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Excercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Try to add a new BatchNormalization layer to the Model \n",
    "# (after the Dropout layer) - before or after the ReLU Activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Addendum:\n",
    "\n",
    "* [CNN on CIFAR10](4.3 CIFAR10 CNN.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
